Notebooks/vnx-benchmark-rtt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmark Round Trip time experiment (point to point)\n",
    "This notebook will show you how to measure the round trip time between two Alveo nodes using the benchmark application with UDP as a transport protocol.\n",
    "We are going to rely on a Dask cluster to configure the local and remote Alveo cards.\n",
    "\n",
    "This notebook assumes:\n",
    "* Direct connection between the Alveo cards\n",
    "* Dask cluster is already created and running. For more information about setting up a Dask cluster visit the [Dask documentation](https://docs.dask.org/en/latest/setup.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://10.1.212.129:8786</li>\n",
       "  <li><b>Dashboard: </b><a href='http://10.1.212.129:8787/status' target='_blank'>http://10.1.212.129:8787/status</a>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>2</li>\n",
       "  <li><b>Cores: </b>32</li>\n",
       "  <li><b>Memory: </b>232.35 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.1.212.129:8786' processes=2 threads=32, memory=232.35 GB>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dask.distributed import Client\n",
    "\n",
    "client = Client(\"tcp://10.1.212.129:8786\")\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client_info = client.scheduler_info()['workers']\n",
    "workers = []\n",
    "for cli in client_info:\n",
    "    workers.append(client_info[cli]['name'])\n",
    "\n",
    "if len(workers) != 2:\n",
    "    print(\"Configure your Dask cluster with two workers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic remote functions\n",
    "In this part we are going to schedule a basic function to the workers to verify that we are able to pinpoint tasks to a particular worker, we are also going to grab the Alveo shell name.\n",
    "You should visually check that your xclbin file is built for the Alveo shell available on the workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Worker name: alveo4b | shell version: \"xilinx_u280_xdma_201920_3\"\n",
      "Worker name: alveo4c | shell version: \"xilinx_u280_xdma_201920_3\"\n"
     ]
    }
   ],
   "source": [
    "import platform, os\n",
    "\n",
    "def verify_workers():\n",
    "    node_name = platform.node()\n",
    "    shell_version = os.popen(\"xbutil dump | grep dsa_name\").read()\n",
    "    #match = True\n",
    "    #if 'xilinx_u280_xdma_201920_3' not in shell_version:\n",
    "    #    match = False\n",
    "    return node_name, shell_version[24:-2]\n",
    "\n",
    "worker_0 = client.submit(verify_workers ,workers=workers[0], pure=False)\n",
    "worker_1 = client.submit(verify_workers ,workers=workers[1], pure=False)\n",
    "\n",
    "worker_check = [worker_0.result(),worker_1.result()]\n",
    "\n",
    "for w in worker_check:\n",
    "    print('Worker name: {} | shell version: {}'.format(w[0],w[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Source Dask device and utilities\n",
    "\n",
    "In this section we will declare the Dask code that builds on top of the `pynq` framework. This piece of code allow us to:\n",
    "\n",
    "* Download a `xclbin` file to a worker\n",
    "* Peek and poke registers\n",
    "* Allocate buffers\n",
    "* Start kernels\n",
    "\n",
    "All of these capabilities are available for both local and remote workers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "try {\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n",
       "} catch (e) {};\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from vnx_utils import *\n",
    "import pynq\n",
    "import ctypes\n",
    "import tempfile\n",
    "import numpy as np\n",
    "import re\n",
    "from pynq.pl_server.xrt_device import XrtStream\n",
    "\n",
    "# Hold references to buffers to avoid them being collected\n",
    "# Won't be visible in the process but is an easy way to\n",
    "# let workers hold on to local references\n",
    "buffers = []\n",
    "\n",
    "# Functions that will be called in the context of dask\n",
    "def _invalidate(bo, offset, size):\n",
    "    buf = bytearray(size)\n",
    "    pynq.Device.active_device.invalidate(bo, offset, 0, size)\n",
    "    pynq.Device.active_device.buffer_read(bo, offset, buf)\n",
    "    return bytes(buf)\n",
    "\n",
    "def _flush(bo, offset, size, data):\n",
    "    pynq.Device.active_device.buffer_write(bo, offset, bytearray(data))\n",
    "    pynq.Device.active_device.flush(bo, offset, 0, size)\n",
    "    \n",
    "def _read_registers(address, length):\n",
    "    return pynq.Device.active_device.read_registers(address, length)\n",
    "\n",
    "def _write_registers(address, data):\n",
    "    pynq.Device.active_device.write_registers(address, data)\n",
    "    \n",
    "def _download(bitstream_data):\n",
    "    with tempfile.NamedTemporaryFile() as f:\n",
    "        f.write(bitstream_data)\n",
    "        f.flush()\n",
    "        ol = pynq.Overlay(f.name)\n",
    "\n",
    "def _alloc(size, memdesc):\n",
    "    mem = pynq.Device.active_device.get_memory(memdesc)\n",
    "    buf = mem.allocate((size,), 'u1')\n",
    "    buffers.append(buf)\n",
    "    return buf.bo, buf.device_address\n",
    "\n",
    "class DaskMemory:\n",
    "    \"\"\"Memory object proxied over dask\n",
    "    \n",
    "    \"\"\"\n",
    "    def __init__(self, device, desc):\n",
    "        self._desc = desc\n",
    "        self._device = device\n",
    "    \n",
    "    def allocate(self, shape, dtype):\n",
    "        from pynq.buffer import PynqBuffer\n",
    "        buf = PynqBuffer(shape, dtype, device_address=0,\n",
    "                         bo=0, device=self._device, coherent=False)\n",
    "        bo, addr = self._device._call_dask(_alloc, buf.nbytes, self._desc)\n",
    "        buf.bo = bo\n",
    "        buf.device_address = addr\n",
    "        return buf\n",
    "\n",
    "class DaskDevice(pynq.Device):\n",
    "    \"\"\"PYNQ Proxy device for using PYNQ via dask\n",
    "    \n",
    "    \"\"\"\n",
    "    def __init__(self, client, worker):\n",
    "        \"\"\"The worker ID should be unique\n",
    "        \n",
    "        \"\"\"\n",
    "        super().__init__(\"dask-\" + re.sub(r'[^\\w]', '_', worker))\n",
    "        self._dask_client = client\n",
    "        self._worker = worker\n",
    "        self.capabilities = {\n",
    "            'REGISTER_RW': True,\n",
    "            'CALLABLE': True\n",
    "        }\n",
    "        self._streams = {}\n",
    "        \n",
    "    def _call_dask(self, func, *args):\n",
    "        future = self._dask_client.submit(func, *args, workers=self._worker, pure=False)\n",
    "        return future.result()\n",
    "\n",
    "    def invalidate(self, bo, offset, ptr, size):\n",
    "        ctype = ctypes.c_uint8 * size\n",
    "        target = ctype.from_address(ptr)\n",
    "        target[:] = self._call_dask(_invalidate, bo, offset, size)\n",
    "        \n",
    "    def flush(self, bo, offset, ptr, size):\n",
    "        ctype = ctypes.c_uint8 * size\n",
    "        target = ctype.from_address(ptr)\n",
    "        self._call_dask(_flush, bo, offset, size, bytes(target))\n",
    "        \n",
    "    def read_registers(self, address, length):\n",
    "        return self._call_dask(_read_registers, address, length)\n",
    "    \n",
    "    def write_registers(self, address, data):\n",
    "        self._call_dask(_write_registers, address, bytes(data))\n",
    "        \n",
    "    def get_bitfile_metadata(self, bitfile_name):\n",
    "        return pynq.pl_server.xclbin_parser.XclBin(bitfile_name)\n",
    "    \n",
    "    def download(self, bitstream, parser=None):\n",
    "        with open(bitstream.bitfile_name, 'rb') as f:\n",
    "            bitstream_data = f.read()\n",
    "        self._call_dask(_download, bitstream_data)\n",
    "        super().post_download(bitstream, parser)\n",
    "    \n",
    "    def get_memory(self, desc):\n",
    "        if desc['streaming']:\n",
    "            if desc['idx'] not in self._streams:\n",
    "                self._streams[desc['idx']] = XrtStream(self, desc)\n",
    "            return self._streams[desc['idx']]\n",
    "        else:\n",
    "            return DaskMemory(self, desc)\n",
    "    \n",
    "    def get_memory_by_idx(self, idx):\n",
    "        for m in self.mem_dict.values():\n",
    "            if m['idx'] == idx:\n",
    "                return self.get_memory(m)\n",
    "        raise RuntimeError(\"Could not find memory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download xclbin to workers\n",
    "1. Create Dask device for each worker\n",
    "2. Create an overlay object for each worker, this step will download the `xclbin` file to the Alveo card"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/tools/external/anaconda/envs/pynq-dask/lib/python3.7/site-packages/distributed/worker.py:3321: UserWarning: Large object of size 60.94 MB detected in task graph: \n",
      "  (b'xclbin2\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff ... ROR_DATA_END',)\n",
      "Consider scattering large objects ahead of time\n",
      "with client.scatter to reduce scheduler burden and \n",
      "keep data on workers\n",
      "\n",
      "    future = client.submit(func, big_data)    # bad\n",
      "\n",
      "    big_future = client.scatter(big_data)     # good\n",
      "    future = client.submit(func, big_future)  # good\n",
      "  % (format_bytes(len(b)), s)\n"
     ]
    }
   ],
   "source": [
    "daskdev_w0 = DaskDevice(client, workers[0])\n",
    "daskdev_w1 = DaskDevice(client, workers[1])\n",
    "\n",
    "xclbin = '../benchmark.intf3.xilinx_u280_xdma_201920_3/vnx_benchmark_if3.xclbin'\n",
    "ol_w0 = pynq.Overlay(xclbin, device=daskdev_w0)\n",
    "ol_w1 = pynq.Overlay(xclbin, device=daskdev_w1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Link \n",
    "\n",
    "We are going to use the function `linkStatus` that reports if the CMAC is detecting link, which means that the physical connection\n",
    "between the two Alveo cards is established."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Link worker 0 {'cmac_link': True}; link worker 1 {'cmac_link': True}\n"
     ]
    }
   ],
   "source": [
    "print(\"Link worker 0 {}; link worker 1 {}\".format(ol_w0.cmac_1.linkStatus(),ol_w1.cmac_1.linkStatus()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure remote Alveo card\n",
    "\n",
    "1. Set up IP address and MAC address\n",
    "2. Set up connection table\n",
    "3. Launch ARP discovery\n",
    "4. Print out ARP Table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'HWaddr': '00:0a:35:02:9d:0a', 'inet addr': '192.168.0.10', 'gateway addr': '192.168.0.1', 'Mask': '255.255.255.0'}\n",
      "Position   5\tMAC address 00:0a:35:02:9d:e5\tIP address 192.168.0.5\n"
     ]
    }
   ],
   "source": [
    "print(ol_w1.networklayer_1.updateIPAddress('192.168.0.10', debug=True))\n",
    "# 2\n",
    "ol_w1.networklayer_1.sockets[0] = ('192.168.0.5', 62177, 60512, True)\n",
    "ol_w1.networklayer_1.populateSocketTable()\n",
    "#3 \n",
    "ol_w1.networklayer_1.arpDiscovery()\n",
    "#4\n",
    "ol_w1.networklayer_1.readARPTable()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure local Alveo card\n",
    "\n",
    "1. Print out IP and MAC address\n",
    "2. Set up connection table\n",
    "3. Launch ARP discovery\n",
    "4. Print out ARP Table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'HWaddr': '00:0a:35:02:9d:e5', 'inet addr': '192.168.0.5', 'gateway addr': '192.168.0.1', 'Mask': '255.255.255.0'}\n",
      "Position  10\tMAC address 00:0a:35:02:9d:ea\tIP address 192.168.0.10\n"
     ]
    }
   ],
   "source": [
    "print(ol_w0.networklayer_1.getNetworkInfo())\n",
    "#2\n",
    "ol_w0.networklayer_1.sockets[2] = ('192.168.0.10', 60512, 62177, True)\n",
    "ol_w0.networklayer_1.populateSocketTable()\n",
    "#3 \n",
    "ol_w0.networklayer_1.arpDiscovery()\n",
    "#4\n",
    "ol_w0.networklayer_1.readARPTable()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure application\n",
    "\n",
    "* Configure remote benchmark `traffic_generator_1_0` application in `LOOPBACK` mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ol_w1_tg = ol_w1.traffic_generator_1_0\n",
    "ol_w1_tg.register_map.mode = benchmark_mode.index('LOOPBACK')\n",
    "ol_w1_tg.register_map.dest_id = 0 # Use connection in position 0 to reflect\n",
    "ol_w1_tg.register_map.CTRL.AP_START = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure local benchmark application\n",
    "This part configures the collector, in particular\n",
    "* Allocate buffers\n",
    "* Start collector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "send_packets   = 2 ** 20\n",
    "shape          = (send_packets,1)\n",
    "rtt_cycles     = pynq.allocate(shape, dtype=np.uint32, target=ol_w0.HBM0)\n",
    "pkt            = pynq.allocate(1,     dtype=np.uint32, target=ol_w0.HBM0)\n",
    "\n",
    "collector_h = ol_w0.collector_1_2.start(rtt_cycles,pkt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**This part configures the traffic generator** `traffic_generator_1_2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "send_pkts = send_packets\n",
    "ol_w0_tg = ol_w0.traffic_generator_1_2\n",
    "ol_w0_tg.register_map.debug_reset = 1\n",
    "ol_w0.networklayer_1.register_map.debug_reset_counters = 1\n",
    "ol_w0_tg.register_map.mode = benchmark_mode.index('LATENCY')\n",
    "ol_w0_tg.register_map.number_packets = send_pkts\n",
    "ol_w0_tg.register_map.time_between_packets = 50\n",
    "ol_w0_tg.register_map.number_beats = 1\n",
    "ol_w0_tg.register_map.dest_id = 2\n",
    "ol_w0_tg.register_map.CTRL.AP_START = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read latency result\n",
    "* Call the dask method to synchronize the Alveo buffer with the dask buffer\n",
    "\n",
    "Note that this buffer contains the round trip time in clock cycles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PynqBuffer([[309],\n",
       "            [308],\n",
       "            [307],\n",
       "            ...,\n",
       "            [308],\n",
       "            [306],\n",
       "            [308]], dtype=uint32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rtt_cycles.invalidate()\n",
    "rtt_cycles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute some statistics on the results\n",
    "1. Convert the rtt from cycles to microseconds, for this particular implementation the clock frequency is 300 MHz. Check current kernel frequency `xbutil | dump`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = 300\n",
    "rtt_usec = np.array(shape, dtype=np.float)\n",
    "rtt_usec= rtt_cycles / freq  # convert to microseconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Use `scipy` to compute statistical values\n",
    "    * Mean\n",
    "    * Standard deviation\n",
    "    * Mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Round trip time at application level using 1,048,576 packets\n",
      "\tmean    = 1.054 us\n",
      "\tstd_dev = 0.003674 us\n",
      "\tmode    = 1.055 us, which appears 378,733 times\n",
      "\tmax     = 1.116 us\n",
      "\tmin     = 1.045 us\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "mean, std_dev, mode = np.mean(rtt_usec), np.std(rtt_usec), stats.mode(rtt_usec)\n",
    "print(\"Round trip time at application level using {:,} packets\".format(len(rtt_usec)))\n",
    "print(\"\\tmean    = {:.3f} us\\n\\tstd_dev = {:.6f} us\".format(mean,std_dev))\n",
    "print(\"\\tmode    = {:.3f} us, which appears {:,} times\".format(mode[0][0][0],mode[1][0][0]))\n",
    "print(\"\\tmax     = {:.3f} us\".format(np.max(rtt_usec)))\n",
    "print(\"\\tmin     = {:.3f} us\".format(np.min(rtt_usec)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Box and whisker graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'whiskers': [<matplotlib.lines.Line2D at 0x7fca8326ead0>,\n",
       "  <matplotlib.lines.Line2D at 0x7fca83276c90>],\n",
       " 'caps': [<matplotlib.lines.Line2D at 0x7fca832c31d0>,\n",
       "  <matplotlib.lines.Line2D at 0x7fca83251d90>],\n",
       " 'boxes': [<matplotlib.lines.Line2D at 0x7fca8326e790>],\n",
       " 'medians': [<matplotlib.lines.Line2D at 0x7fca83276d10>],\n",
       " 'fliers': [<matplotlib.lines.Line2D at 0x7fca8327f990>],\n",
       " 'means': []}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABAkAAACqCAYAAADPwBbSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAW3ElEQVR4nO3debTkZX3n8feHbhSQfZEdWyIquLAMi4Zd1BFNMjqjiEENJpFjBobIjBkzOUYxxx2jCeq4ELHFOAxmBgkiRlChBUOrzdoIMRKGTaABAbGhEej+zh+/5ybFpe7t23ftqvt+nVOnq+q3PN/f77l1b9ennudXqSokSZIkSZI2mOsCJEmSJEnS+sGQQJIkSZIkAYYEkiRJkiSpMSSQJEmSJEmAIYEkSZIkSWoMCSRJkiRJEmBIIEnSvJFkUZJKsnCK+6kkzxlj2XFJLprAPhYn+cBU6pisJJcm+cO5aFuSpPWdIYEkaWgkuSXJqiQrk9zd3ohu2pZ9qz2/MsnjSR7refyznvurkqzpebxyjLYqycNtnfuSnJ1ky9k94vVPVX21ql4513VMh+kKVSRJGiSGBJKkYfPbVbUpsA+wL/A/AKrq6KratC37KvCxkcdVtUfPsqOBO3uWbTpOW3u35bsDWwGnzuSB6al8Ay9J0vQyJJAkDaWquhv4Nl1YMNNtPQScD+w18lySnZKcn+T+JDcleXvPsguT/GXP43OSnNlv30kOTHJFkgeT3JXk00me1rO8kryjjYZ4IMlnkqQtW5Dk422kw83Aa8Y6hiRvS/KNnsc3Jflaz+Pbk/Sey5eP0ebxSS5v95Pkk0nuSfLLJNcleWGftjdLckmS09s2T29135ZkRZLPJdm4rXtEkjuSvDvJ3cCX+uzv+CQ/SPKp1u4/JTlqjOPeIMl7ktza6jwryRZt8ffbvw+2ESMvHev8SZI0LAwJJElDKckudKMCbpqFtrYCXgss7Xn6bOAOYCfg9cCHet6o/j7wliQvS3IccADwx2PsfjVwCrAt8FLgKOA/j1rnt9o+9gaOAf59e/7tbdm+wP6tjrEsAQ5tb5p3BDYEDm7HtzuwKXDdBNrs9UrgMOC5wJbAG4Ff9K6QZBvgu8APqurkqirgo22bfYDnADsD7+3ZbAdga+BZwAljHM9BwM105+19wLlJtu6z3vHtdiTdiJBNgU+3ZYe1f7dso0quGKMtSZKGhiGBJGnYnJfkV8DtwD10bxBnylVJHgTuA3YDPg+QZFfgEODdVfVoVV0D/A3wFvjXUQ7vAL4M/DXw1qr6Vb8GqurKqlpaVU9U1S2tjcNHrfaRqnqwqm4DLuHfRk8cA/xVVd1eVfcDHx7rQKrqZuBXbdvD6UZh/DzJ89vjy6pqzQTa7PU4sBnwfCBVdWNV3dWzfCe6cOLvquo97dyFLtw4parub+flQ8CxPdutAd5XVb+uqlVjHNI97dgfr6pzgJ/SfyTFccAnqurmqlpJNz3lWKcxSJLmK0MCSdKweW1VbQYcQffmdNsZbGu/qtoS2Aj4LHBZko3o3vzeP+qN/610n4iPuABYAPy0qi4fq4Ekz01yQboLMT5E94Z59DHd3XP/EbpPw2l13D6qhvEsoTtvh7X7l9IFBIe3xxNp819V1ffoPpX/DLAiyReSbN6zymuAjYHP9Ty3HbAJcGWbYvEg8A/t+RH3VtWjazmWn7dRCSNupTsfo+3Ek8/LrcBCYPu17F+SpKFkSCBJGkpVtQRYDHx8Ftp6nG6kwLOBFwJ3Alsn2axntd2An/c8/iBwI7BjkjeNs/vPAv8E7FFVmwN/BmSCpd0F7DqqhvGMhASHtvtLGDskmJCqOr2q/h3wAropBH/Ss/gMugDgwiTPaM/dB6wCXlBVW7bbFqMuINn75n8sO49cJ6HZja5fRruTbtpC73pPACsm2I4kSUPFkECSNMz+CnjFqAvuTbskC4C30b25vbmqbgf+Efhwko2SvBj4A7pvVSDJYW39t7bbp5Ls3Hfn3XD9h4CVbej/H61DaV8DTk6yS7tuwp+uZf0ldHPzN66qO4DLgFcB2wBXr0O7ACQ5IMlBSTYEHgYepbvGQq+T6KYCXJBk4zal4Qzgk0me2fazc5J+1zwYzzPpjn3DJG8A9gQu7LPe2cApSZ6d7usyPwScU1VPAPfSTW3YfR3bliRpYBkSSJKGVlXdC5wF/PkMNXFtkpXAA8DvAa9rc/8B3gQsovuk+ut0c+gvbsPtzwJOqqqft6kGXwS+NOqT7xHvAn6X7noBZwDnrEN9Z9BdW+Ba4Crg3PFWrqp/BlbShQMj39pwM91FBUe/uZ+IzVsND9AN4/8Fo0Z2tCkBJ9BNi/j7Nl3j3XQXnFzaplh8B3jeOrb9Q2APupEJHwReX1W/6LPemcBX6L7J4P/RBRn/pdX2SNv2B23qw0vWsQZJkgZOnjxdT5IkabAlOR74w6o6ZK5rkSRp0DiSQJIkSZIkAYYEkiRJkiSpcbqBJEmSJEkCHEkgSZIkSZIaQwJJkiRJkgTAwnVZedttt61FixbNUCmSJEmSJGkyrrzyyvuqarup7medQoJFixaxbNmyqbYpSZIkSZKmUZJbp2M/TjeQJEmSJEmAIYEkSZIkSWoMCSRJkiRJEmBIIEmSJEmSGkMCSZIkSZIEGBJIkiRJkqTGkECSJEmSJAGGBJIkSZIkqTEkkCRJkiRJgCGBJEmSJElqDAkkSZIkSRJgSCBJkiRJkhpDAkmSJEmSBBgSSJIkSZKkxpBAkiRJkiQBhgSSJEmSJKkxJJAkSZIkSYAhgSRJkiRJagwJJEmSJEkSYEggSZIkSZIaQwJJkiRJkgQYEkiSJEmSpMaQQJIkSZIkAYYEkiRJkiSpMSSQJEmSJEmAIYEkSZIkSWoMCSRJkiRJEmBIIEmSJEmSGkMCSZIkSZIEGBJIkiRJkqTGkECSJEmSJAGGBJIkSZIkqTEkkCRJkiRJgCGBJEmSJElqDAkkSZIkSRJgSCBJkiRJkhpDAkmSJEmSBBgSSJIkSZKkxpBAkiRJkiQBhgSSJEmSJKkxJJAkSZIkSYAhgSRJkiRJagwJJEmSJEkSYEggSZIkSZIaQwJN2tZbb02Sgblx6hZzfcokSZIkab1mSKBJe+CBB6iqgblJkiRJksZnSCBJkiRJkgBDAkmSJEmS1BgSrGeSzHUJWg/4cyBJkiRpLgxlSLDxGBeu29g3XtK0GaTX2SDVCoNV7yDVCoNXrzRoFu2wQ9/X2KIddpjr0voapN8Jg1QrDFa91jpzBqneZyxY0LfWZyxYMNelzbqFc13ATHgU6HeZuvXvR1EaXIP0OhukWmGw6h2kWmHw6pUGza0rVvR/ja1YMeu1TMQg/U4YpFphsOq11pkzSPU+smZN/1rXrJn1WubaUI4kkCRJkiRJ626tIwmSnACcALDbbrvNeEFyPvpM8txKkiRJ0tjWGhJU1ReALwDsv//+ftn8LKgajNM8iG+4PbeSJEmSNDanG0iSJEmSJGBIL1y4Ef0vhrHRbBciDbFBep0NUq0wWPUOUq0wePVKg+ZZ22/f9yKFz9p++zmoZu0G6XfCINUKg1Wvtc6cQap3kw026HuRwk02mH+fqw9lSLBqQIaUS4NskF5ng1QrDFa9g1QrDF690qC55e6757qEdTJIvxMGqVYYrHqtdeYMUr0Pr1491yWsN+ZfLCJJkiRJkvoyJFjPDMqF9TSz/DmQJEmSNBcMCSRJkiRJEmBIIEmSJEmSGkMCTUmSgblJkiRJksY3lN9uoNnhvHlJkiRJGi6OJJAkSZIkSYAhgSRJkiRJagwJJEmSJEkSYEggSZIkSZIaQwJJkiRJkgQYEkiSJEmSpMaQQJIkSZIkAYYEkiRJkiSpMSSQJEmSJEmAIYEkSZIkSWoMCSRJkiRJEmBIIEmSJEmSGkMCSZIkSZIEGBJIkiRJkqTGkECSJEmSJAGGBJIkSZIkqTEkkCRJkiRJgCGBJEmSJElqDAkkSZIkSRJgSCBJkiRJkhpDAkmSJEmSBBgSSJIkSZKkxpBAkiRJkiQBhgSSJEmSJKkxJJAkSZIkSYAhgSRJkiRJagwJJEmSJEkSYEggSZIkSZIaQwJJkiRJkgQYEkiSJEmSpMaQQJIkSZIkAYYEkiRJkiSpMSSQJEmSJEmAIYEkSZIkSWoMCSRJkiRJEmBIIEmSJEmSGkMCSZIkSZIEGBJIkiRJkqTGkECSJEmSJAGGBJIkSZIkqTEkkCRJkiRJgCGBJEmSJElqDAkkSZIkSRJgSCBJkiRJkppU1cRXTu4Fbp25cjSLtgXum+siNCfs+/nLvp+/7Pv5y76fv+z7+cu+n7+eV1WbTXUnC9dl5arabqoNav2QZFlV7T/XdWj22ffzl30/f9n385d9P3/Z9/OXfT9/JVk2HftxuoEkSZIkSQIMCSRJkiRJUmNIMH99Ya4L0Jyx7+cv+37+su/nL/t+/rLv5y/7fv6alr5fpwsXSpIkSZKk4eVIAkmSJEmSBBgSDJ0kZya5J8n1YyxPktOT3JTkuiT79SxbneSadjt/9qrWdJhi3++W5KIkNya5Icmi2apbUzfZvk9yZM9r/pokjyZ57exWr6mY4uv+Y0l+0l73pyfJ7FWuqZpi3380yfXt9sbZq1rTYQJ9//wkVyT5dZJ3jVr2qiQ/bT8Xfzo7FWu6TLHvx91W67fJ9n2SXZNc0v7W/yTJH0+kPUOC4bMYeNU4y48G9mi3E4DP9ixbVVX7tNvvzFyJmiGLmXzfnwWcVlV7AgcC98xQjZoZi5lE31fVJSOveeBlwCPARTNbqqbZYibR90l+EzgYeDHwQuAA4PCZLFTTbjGT6/vXAPsB+wAHAX+SZPMZrVTTbTHj9/39wMnAx3ufTLIA+Azdz8ZewJuS7DVDNWpmLGYSfT/BbbV+W8zk+v4J4L+1/+O/BDhxIq97Q4IhU1Xfp/shGct/AM6qzlJgyyQ7zk51mkmT7fv2i2JhVV3c9rOyqh6ZhZI1Tabpdf964Fv2/WCZQt8XsBHwNODpwIbAipmuV9NnCn2/F7Ckqp6oqoeBa/GNw0BZW99X1T1V9WPg8VGLDgRuqqqbq+ox4H/T/ZxoQEyh7yfyO0Prscn2fVXdVVVXtfu/Am4Edl5be4YE88/OwO09j+/g335QNkqyLMlShxwPpbH6/rnAg0nOTXJ1ktPapw0aHuO97kccC5w9axVptvTt+6q6ArgEuKvdvl1VN85BfZo5Y73urwWOTrJJkm2BI4Fd56A+zb6J/C2QNKTadOJ9gR+ubd2FM12M1jv95pyOfMXFblV1Z5Ldge8lWV5V/zKLtWlmjdX3C4FD6X5p3AacAxwPfHHWKtNMG+91T/t08UXAt2etIs2Wvn2f5DnAnsAu7bmLkxzWPqnQcOjb91V1UZIDgH8E7gWuoBuOquE37t8CScMryabA/wXeWVUPrW19RxLMP3fw5E8MdgHuBKiqkX9vBi6le9Oo4TFW398BXN2GHz4BnEc3X1XDY8zXfXMM8PWqesrwRA28sfr+dcDSNr1oJfAturmKGh7j/b3/YLseySvo3jj+bA7q0+xb298CSUMoyYZ0AcFXq+rciWxjSDD/nA+8tV31+CXAL6vqriRbJXk6QBt+eDBww1wWqmnXt++BHwNbJdmurfcy7PthM1bfj3gTTjUYVmP1/W3A4UkWtv88HE43T1HDY6y/9wuSbAOQ5MV0F6/0gqXzw4+BPZI8O8nT6KaZ+W1W0hBr31z0ReDGqvrERLdzusGQSXI2cASwbZI7gPfRXZCKqvoccCHwauAmuiuZv61tuifw+SRr6MKjj1SVbxQHyGT7vqpWt69K+W77RXIlcMasH4AmbQqv+5H5absCS2azZk2PKfT9/6ELBJfTDTf+h6r6xqwWrymZQt9vCFzW/brnIeDNbRSZBsTa+j7JDsAyYHNgTZJ3AntV1UNJTqKbWrYAOLOqfjIXx6DJmWLfP2XbqnJq6YCYbN/TBcFvAZYnuabt7s+q6sJx26tyKpIkSZIkSXK6gSRJkiRJagwJJEmSJEkSYEggSZIkSZIaQwJJkiRJkgQYEkiSJEmSpMaQQJI0FJKsTnJNkuuTfCPJlrPQ5i1Jth313A9bHbclubfdv6Z93eTo7b+U5HkTbOu9Pfta3XP/xHY7bnqO6intfj3Js9Zh/YOSfHImapltSRYmebDd3yHJN+e6JkmSZppfgShJGgpJVlbVpu3+l4F/rqoPznCbtwD7V9V9fZYd35adNMa2C6pq9STaXAjcV1WzEYLsDbynqt4wg21M6jzMhtHnOslXgE9X1Q/ntjJJkmaOIwkkScPoCmBngHROayMMlid5Y3v+iCQXjGyQ5NPtjf3ICIH3J7mqbfP89vw2SS5KcnWSzwOZaEEjn0on+UCSHwEHJrk8yT49yz7Z2rw4yTbrsO8PJHlnu395kk8kuSzJDUn2b6MBfpbk1J5tfi/Jj9pohP+ZpN//CY4D/n5U/ae1Gr/dRg0sSXJzkle39V6e5Lx2f7MkX27n8Lokrx3jPLyi1bE8yRlJnta2P60dw3VJPtqe2z7JuUmWtfpfMlZb7fk3t+euT/KhUcfykSTXJrkiyTPbst9INxrkx8CpPNl57ZxIkjS0DAkkSUMlyQLgKOD89tR/BPYB9gZeDpyWZMcJ7Oq+qtoP+Czwrvbc+4DLq2rftv/d1rG8LYCrqurAqrqiz7Klrc0rgD9fx333WlVVhwJfpHtj+w7gRcAJSbZM8kLgdcBvVtU+wELg2D77ORi4clSNF7UaH6N7E30U8AbgL/psfypwb1W9iO78L+nZz1VVdSBwLXAm8J/aepu0OrcHXg28oKpeDHy4bXs68LGq2h84BvibsdpKsgvwAeBIYF/g4CS/1VPDkqram+58/357/lPAX1fVAcC9o45nGXBon+OUJGloGBJIkobFxkmuAX4BbA1c3J4/BDi7qlZX1Qq6N6oHTGB/57Z/rwQWtfuHAX8LUFXfBB5YxxofA74+xrIngL9r9/+21T1ZIwHJcmB5Va2oqkeBW4Bd6MKSA4Bl7ZwdDvxGn/3syJPfKK+qqpHzuhy4tKqeaPcX9dn+5cBnAKozcr56z8OewM+q6l/a47PozvP9wBrgjCSvAx7u2efnWt3nAVsl2XiMtg4CvldV91XV48D/avseOZZvtfu9ffxS4Jx2/yujjuceYKc+xylJ0tBYONcFSJI0TVZV1T5JtgAuAE6k+9R5rCkBT/DksHyjUct/3f5dzZP/Xk7lYj6rauyLAY1+firtjNS+puf+yOOFdOfkzKpa22iFVTz5vDw2al+97fT7P0Xofxy956Fv/1TV40n2B15BN8rhj4BXtvUPrKreWkjSr63xpoP0bt/bxzVGzdCdi1Xj7FOSpIHnSAJJ0lCpql8CJwPvSrIh8H3gjUkWJNmO7pPkHwG3AnsleXoLFo6awO6/T5uTnuRoYKtpLH1DuqkRAL8LXD6N+x7tO8Axad/M0K610G/qxI3Ac6bQzkXASa2NJOl3vm4A9kiye3v8ZrqpApsBm1fVBcApdNMFRmo/cWTjJPuM09ZS4Mh2fCNTKkamPIxlKd00Bnjq9QeeC1y/lu0lSRpohgSSpKFTVVfTzXU/lm5Y+3Xt8feA/15Vd1fV7cDX2rKvAldPYNfvBw5LchXdp9q3TWPZvwT2a/s+hG4u/YyoquV0x/KdJNfRvcHevs+q3wSOmEJT7we2T3I9cA195vNX1SPAHwDnJllONzrhDLprBnwzyUi//de2yYl01xa4LskNwNvHaquq7gDeC1zanlvapomM52TglHZRxU1HLTuS7pxIkjS0/ApESZLmWGbxaw3XRZJNgO8Ch6yvX1M4W9p0hsuA17TRKpIkDSVHEkiSpL7ap/x/QXcBw/numXTfqmBAIEkaao4kkCRJkiRJgCMJJEmSJElSY0ggSZIkSZIAQwJJkiRJktQYEkiSJEmSJMCQQJIkSZIkNYYEkiRJkiQJgP8PE/aUNNFytHYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1296x144 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "red_square = dict(markerfacecolor='r', marker='s')\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_title('RTT Box and whisker plot')\n",
    "ax.set_xlabel('Round Trip Time (microsecond)')\n",
    "ax.set_yticklabels([''])\n",
    "fig.set_size_inches(18, 2)\n",
    "ax.boxplot(rtt_usec, vert=False, flierprops=red_square)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Release Alveo cards\n",
    "* To release the alveo cards the pynq overlay is freed\n",
    "* Delete dask pynq-dask buffers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pynq.Overlay.free(ol_w0)\n",
    "pynq.Overlay.free(ol_w1)\n",
    "del rtt_cycles\n",
    "del pkt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------\n",
    "Copyright (c) 2020-2021, Xilinx, Inc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}