Scenario_Coverage_A9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario K-projection Metric for German Traffic Highway\n",
    "\n",
    "In this metric, we are interested in understanding the completeness of the test data (more generally, it can also be used to measure the completeness of training data). Apart from a random & unsystematic approach to drive as many kilometers as possible, we have carefully selected a segment of A9 in order to create as many variations as possible for the same data.\n",
    "\n",
    "The road segment being selected is from **Munuch Schwabing** to **Munich Freimann**. By observing the map below, one can see that in this short segment, we encounter left curves, right curves, and straight lines. \n",
    "\n",
    "The completeness of data plays an important role in terms of understanding the performance of the network. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, the following imports are standard. Be sure to have required libraries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put these at the top of every notebook, to get automatic reloading and inline plotting\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import cv2\n",
    "from scipy.misc import imread, imresize\n",
    "from matplotlib import pyplot as plt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![German Highway A9 from Schwabing to Freimann](img/A9_Schwabing_Freimann.jpg?)\n",
    "![Variations](img/Variations.jpg?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above 6 images shows the vehicle driving on the same road. They are under different lighting conditions (day, night), under different road surface situations (dry, wet), under different lanes (inner, outer), under different weathers (sunny, cloudy, rainy). One can easily observe that a complete data sampling will require exponentially many videos taken. In this simple condition, even for the same place, we already need 2x2x2x3=24 videos. \n",
    "\n",
    "If you consider the road curving (left, straight, right), the total number of lanes (2, 3, 4), as well as additional weather conditions such as snowy, the orientation of the sun (right, back, left), even in such a small segment, one can easily needs up 1000 videos for a systematic testing, even on such a small segment. \n",
    "\n",
    "The below demonstrated **scenario k-projection metric** tries to create a weaker form of completeness which avoids combinatorial explosion while still guaranteeing *diversity* of test data. The diversity of the test data can later be used also for understanding the effects of physical transformation. More importantly, **the data completeness demonstrated by the diversity and proven by a science-driven metric allows certification authority to believe that the data sampling is approached in a displined manner**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following *description* file sepcifies all possible categorizations under consideration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nndependability.metrics import ScenarioKProjection\n",
    "\n",
    "metric2 = ScenarioKProjection.Scenario_KProjection_Metric(\"data/scenario_coverage/A9/description.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below file specifies restrictions where certain combination is not possible, for the A9 Schwabing to Freimann segment. We refer readers to look at the file, which contains detailed explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0 <=  + 1.0 C0_4 <= 0.0\n",
      "0.0 <=  + 1.0 C1_1 + 1.0 C0_2 <= 1.0\n",
      "0.0 <=  + 1.0 C1_1 + 1.0 C0_3 <= 1.0\n",
      "0.0 <=  + 1.0 C2_0 + 1.0 C3_1 <= 1.0\n",
      "0.0 <=  + 1.0 C2_0 + 1.0 C3_2 <= 1.0\n",
      "0.0 <=  + 1.0 C2_0 <= 0.0\n",
      "0.0 <=  + 1.0 C2_2 + 1.0 C4_1 <= 1.0\n",
      "0.0 <=  + 1.0 C2_2 + 1.0 C4_0 <= 1.0\n",
      "0.0 <=  + 1.0 C2_2 + 1.0 C4_3 <= 1.0\n",
      "0.0 <=  + 1.0 C2_2 + 1.0 C4_4 <= 1.0\n",
      "0.0 <=  + 1.0 C2_1 + 1.0 C3_2 <= 1.0\n"
     ]
    }
   ],
   "source": [
    "metric2.addDomainRestrictionsFromFile(\"data/scenario_coverage/A9/domain-restrictions_A9_Freimann.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we load existing scenarios where in each scenario, it has a field stating which video file is it associated (here the video clip is not presented). Then the solver automatically computed the 2 projection coverage (k=2 is by default) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2-projection coverage (without considering domain restrictions): 111/162\n"
     ]
    }
   ],
   "source": [
    "metric2.addScenariosFromFile(\"data/scenario_coverage/A9/scenarios.xml\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The denominator 162 has not considered situations where some scenario combinations are not possible. Actually one can compute the number of cases that are not possible (the tool provides a semi-automatic, iterative way to do that). Based on the computation, actually 42 cases are not possible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above value has demonstrated that we have covered quite a bit, by considering the total number of 120 cases. The question remains to be **\"what is the next sample to be taken, in order to maximally increase coverage?\"**\n",
    "\n",
    "The automatic test case generator (see below) can provide you such an information, which specifies that we need a night video which is cloudy. The road needs to be dry and straight, and it needs to be on the 3 lane drive where the vehicle is on the 2nd lane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nndependability.atg.scenario import scenariogen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal solution found\n",
      "Maximum possibility for improvement = 51\n",
      "Optimal objective value computed from IP = 4\n",
      "\n",
      "for criterion weather, set it to cloudy\n",
      "for criterion day, set it to night\n",
      "for criterion total_lanes, set it to 3\n",
      "for criterion current_lane, set it to 2\n",
      "for criterion straight_curvy, set it to straight\n",
      "for criterion road_surface, set it to dry\n"
     ]
    }
   ],
   "source": [
    "variableAssignment = scenariogen.proposeScenariocandidate(metric2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can store the criterion to a scenario file, and after we add it, we indeed increase the coverage by 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric2.writeScenarioToFile(variableAssignment, \"tmp.xml\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}