ENSF612_Spark_Project_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ![Spark Logo](http://spark-mooc.github.io/web-assets/images/ta_Spark-logo-small.png) + ![Python Logo](http://spark-mooc.github.io/web-assets/images/python-logo-master-v3-TM-flattened_small.png)\n",
    "# **Linear Regression Lab**\n",
    "#### This lab covers a common supervised learning pipeline, using a subset of the Deerfoot Trail data introduced in a previous lab. Our goal is to train a linear regression model to predict Deerfoot commute times given weather and accident conditions\n",
    "#### ** This lab will cover: **\n",
    "   #### *Part 1:* Read and parse the initial dataset\n",
    "   #### *Part 2:* Create and evaluate a baseline model\n",
    "   #### *Part 3:* Train and evaluate a linear regression model\n",
    " \n",
    "#### Note that, for reference, you can look up the details of the relevant Spark methods in [Spark's Python API](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD) and the relevant NumPy methods in the [NumPy Reference](http://docs.scipy.org/doc/numpy/reference/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkContext\n",
    "sc = SparkContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 1: Read and parse the initial dataset **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1a) Load and check the data **\n",
    "#### The raw data is currently stored in 3 text files.  Commute time data is stored in one of these files. This is the same file you used in a previous lab.  Weather data for the duration of the commute time dataset is stored in 2 separate files - one for 2013 and one for 2014.  This data was obtained from Environment Canada.  We will start by storing the commute time data in a RDD and the weather data in 2 separate RDDs.  Each element of these RDDs is a comma separated string.  Use the [count method](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.count) to check how many data points/lines we have in each of the RDDs.  Then use the [take method](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.take) to create and print out a list of the first 2 data points in their initial string format from the commute time RDD and first 15 lines from each of the weather RDDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileName1 = 'deerfoot_part2-1.csv'\n",
    "fileName2 = 'eng-daily-01012013-12312013.csv'\n",
    "fileName3 ='eng-daily-01012014-12312014.csv'\n",
    "deerfootRDD = (sc.textFile(fileName1, 8))\n",
    "weather2013RDD = sc.textFile(fileName2,8)\n",
    "weather2014RDD = sc.textFile(fileName3,8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "202\n",
      "['21/09/2013,Saturday,34,34,34,34,35,34,35,36,38,36,36,35,35,35,35,35,36,34,34,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2', '22/09/2013,Sunday,34,34,34,34,34,34,34,35,35,35,34,35,34,35,34,34,34,34,34,0,0,0,1,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3']\n",
      "391\n",
      "391\n",
      "['\"Station Name\",\"CALGARY INTL A\"', '\"Province\",\"ALBERTA\"', '\"Latitude\",\"51.12\"', '\"Longitude\",\"-114.01\"', '\"Elevation\",\"1099.10\"', '\"Climate Identifier\",\"3031092\"', '\"WMO Identifier\",\"71877\"', '\"TC Identifier\",\"YYC\"', '', '\"Legend\"', '\"A\",\"Accumulated\"', '\"C\",\"Precipitation occurred, amount uncertain\"', '\"E\",\"Estimated\"', '\"F\",\"Accumulated and estimated\"', '\"L\",\"Precipitation may or may not have occurred\"']\n",
      "['\"Station Name\",\"CALGARY INTL A\"', '\"Province\",\"ALBERTA\"', '\"Latitude\",\"51.12\"', '\"Longitude\",\"-114.01\"', '\"Elevation\",\"1099.10\"', '\"Climate Identifier\",\"3031092\"', '\"WMO Identifier\",\"71877\"', '\"TC Identifier\",\"YYC\"', '', '\"Legend\"', '\"A\",\"Accumulated\"', '\"C\",\"Precipitation occurred, amount uncertain\"', '\"E\",\"Estimated\"', '\"F\",\"Accumulated and estimated\"', '\"L\",\"Precipitation may or may not have occurred\"']\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "numPoints = deerfootRDD.count()\n",
    "print(numPoints)\n",
    "samplePoints = deerfootRDD.take(2)\n",
    "print(samplePoints)\n",
    "num2013WeatherLines = weather2013RDD.count()\n",
    "num2014WeatherLines = weather2014RDD.count()\n",
    "print(num2013WeatherLines)\n",
    "print(num2014WeatherLines)\n",
    "sample2013WeatherLines = weather2013RDD.take(15)\n",
    "sample2014WeatherLines = weather2014RDD.take(15)\n",
    "print(sample2013WeatherLines)\n",
    "print(sample2014WeatherLines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1b) Preprocessing RDDs - extracting desired fields from commute time data **\n",
    "#### We need to pre-process the RDDs before we can use them for training a regression model.  For this exercise, we will consider predicting the commute time at 5 PM in the evening when traffic is typically heavy.  As a result we won't be using all the fields in the 'deerfootRDD'.  Specifically, we will only be using the following fields - Date (field0), Day(field1), Commute Time at 5 PM (field15), and Total number of accidents (field46).  Use a Spark transformation to create a new pairRDD whose key is the Date and whose value is the tuple (Day, Commute Time at 5 PM, Total number of accidents). You need to use the function 'extractFields' to achieve this.  Print the first 2 elements of the resulting pairRDD as a check to see if you extracted the correct fields., `u'split,me'.split(',')` returns `[u'split', u'me']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('21/09/2013', ('Saturday', '35', '2')), ('22/09/2013', ('Sunday', '35', '3'))]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def extractFields(deerfootRDDRecord):\n",
    "    \"\"\"Creates a key-value pair consisting of the Date field as the key and the the tuple (Day,Commute Time at 5 PM, Total number of Accidents) as the value\n",
    "\n",
    "    Args:\n",
    "        deerfootRDDRecord : a comma separated string consisting of all fields in the data set.\n",
    "\n",
    "    Returns:\n",
    "        extracted record: key-value pair as detailed above\n",
    "    \"\"\"\n",
    "    fieldsList = deerfootRDDRecord.split(',')\n",
    "    return (fieldsList[0], (fieldsList[1], fieldsList[15], fieldsList[46]))\n",
    "deerfootPairRDD = deerfootRDD.map(extractFields)\n",
    "print(deerfootPairRDD.take(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1c) Preprocessing RDDs - extracting desired lines from weather data **\n",
    "#### If you notice the output of the weather files the first few lines don't contain the data - they contain metadata and column labels.  These need to be removed.  Use a Spark transformation to discard unwanted metadata and header information.  Hint: You can see whether a unicode string string1 contains another string in variable string2 using 'u'string1' in string2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"2013-01-01\",\"2013\",\"01\",\"01\",\"�\",\"1.2\",\"\",\"-8.9\",\"\",\"-3.9\",\"\",\"21.9\",\"\",\"0.0\",\"\",\"0.0\",\"T\",\"0.0\",\"\",\"0.0\",\"T\",\"4\",\"\",\"33\",\"\",\"48\",\"\"', '\"2013-01-02\",\"2013\",\"01\",\"02\",\"�\",\"5.8\",\"\",\"-10.9\",\"\",\"-2.6\",\"\",\"20.6\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"3\",\"\",\"29\",\"\",\"41\",\"\"', '\"2013-01-03\",\"2013\",\"01\",\"03\",\"�\",\"8.8\",\"\",\"-8.8\",\"\",\"0.0\",\"\",\"18.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"2\",\"\",\"28\",\"\",\"52\",\"\"', '\"2013-01-04\",\"2013\",\"01\",\"04\",\"�\",\"2.9\",\"\",\"-11.2\",\"\",\"-4.2\",\"\",\"22.2\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"2\",\"\",\"26\",\"\",\"63\",\"\"', '\"2013-01-05\",\"2013\",\"01\",\"05\",\"�\",\"4.8\",\"\",\"-10.0\",\"\",\"-2.6\",\"\",\"20.6\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"1\",\"\",\"27\",\"\",\"41\",\"\"']\n",
      "['\"2014-01-01\",\"2014\",\"01\",\"01\",\"�\",\"-0.6\",\"\",\"-9.7\",\"\",\"-5.2\",\"\",\"23.2\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"1.0\",\"\",\"0.8\",\"\",\"9\",\"\",\"\",\"\",\"<31\",\"\"', '\"2014-01-02\",\"2014\",\"01\",\"02\",\"�\",\"8.5\",\"\",\"-10.0\",\"\",\"-0.8\",\"\",\"18.8\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"11\",\"\",\"27\",\"\",\"52\",\"\"', '\"2014-01-03\",\"2014\",\"01\",\"03\",\"�\",\"3.0\",\"\",\"-11.0\",\"\",\"-4.0\",\"\",\"22.0\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"7.8\",\"\",\"5.8\",\"\",\"8\",\"\",\"36\",\"\",\"54\",\"\"', '\"2014-01-04\",\"2014\",\"01\",\"04\",\"�\",\"-11.0\",\"\",\"-21.9\",\"\",\"-16.5\",\"\",\"34.5\",\"\",\"0.0\",\"\",\"0.0\",\"\",\"1.0\",\"\",\"0.2\",\"\",\"14\",\"\",\"35\",\"\",\"44\",\"\"', '\"2014-01-05\",\"2014\",\"01\",\"05\",\"�\",\"-14.4\",\"\",\"-25.5\",\"\",\"-20.0\",\"\",\"38.0\",\"\",\"0.0\",\"\",\"0.0\",\"T\",\"0.0\",\"\",\"0.0\",\"T\",\"14\",\"\",\"36\",\"\",\"52\",\"\"']\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def filterLines(weatherRDDRecord):\n",
    "    \"\"\" Skips lines with metadata and label information\n",
    "    Args:\n",
    "        weatherRDDRecord : a line from the weather file.\n",
    "\n",
    "    Returns:\n",
    "        True - if line is not metadata/label; False otherwise\n",
    "    \"\"\"\n",
    "    fieldsList = weatherRDDRecord.split(',')\n",
    "    if '2013' in fieldsList[0] or '2014' in fieldsList[0]:\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "filteredWeather2013RDD = weather2013RDD.filter(filterLines)\n",
    "filteredWeather2014RDD = weather2014RDD.filter(filterLines)\n",
    "print(filteredWeather2013RDD.take(5))\n",
    "print(filteredWeather2014RDD.take(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1d) Preprocessing RDDs - Fixing date format inconsistency**\n",
    "#### If you compare the Date fields of the commute time and weather files you will notice that they are in different formats.  While the date in the commute time file is in the format \"day/month/year\" it is in the format \"year-month-day\".  We need to fix things so that the weather data has the same date format as the commute time data. (It is very typical to deal with such annoyances while developing models from different data sources). Use a Spark transformation that constructs a pairRDD whose key is the date and whose value is a tuple containing the rest of the columns of the weather data.  Note, you need to construct 2 pairRDDs, one for 2013 and another for 2014.  Also, as mentioned previously, you need to fix the date formatting such that it matches the formatting in the commute time data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('01/01/2013', ['2013', '01', '01', '�', '1.2', '', '-8.9', '', '-3.9', '', '21.9', '', '0.0', '', '0.0', 'T', '0.0', '', '0.0', 'T', '4', '', '33', '', '48', '']), ('02/01/2013', ['2013', '01', '02', '�', '5.8', '', '-10.9', '', '-2.6', '', '20.6', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '3', '', '29', '', '41', ''])]\n",
      "365\n",
      "[('01/01/2014', ['2014', '01', '01', '�', '-0.6', '', '-9.7', '', '-5.2', '', '23.2', '', '0.0', '', '0.0', '', '1.0', '', '0.8', '', '9', '', '', '', '<31', '']), ('02/01/2014', ['2014', '01', '02', '�', '8.5', '', '-10.0', '', '-0.8', '', '18.8', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '11', '', '27', '', '52', ''])]\n",
      "365\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def fixDate(weatherRDDRecord):\n",
    "    \"\"\"Creates a key-value pair - key is date in format 'day/month/year' value - a comma separated string containing other fields of the record\n",
    "    Args:\n",
    "        weatherRDDRecord : a comma separated string consisting of all fields in the weather data set.\n",
    "\n",
    "    Returns:\n",
    "        extracted record: key-value pair as detailed above\n",
    "    \"\"\"\n",
    "    fieldList = weatherRDDRecord.replace('\"','').split(',')\n",
    "    date = fieldList[0].split('-')\n",
    "    return '/'.join(date[::-1]), (fieldList[1:])\n",
    "   \n",
    "fixedWeather2013RDD = filteredWeather2013RDD.map(fixDate)\n",
    "fixedWeather2014RDD = filteredWeather2014RDD.map(fixDate)\n",
    "print(fixedWeather2013RDD.take(2))\n",
    "print(fixedWeather2013RDD.count())\n",
    "print(fixedWeather2014RDD.take(2))\n",
    "print(fixedWeather2014RDD.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1e) Extracting the desired days from weather data**\n",
    "#### commute times have been recorded for the period September 21, 2013 to April 10, 2014.  We need to consider only the weather data for this period while building the linear regression model.  Use a Spark transformation to append the 'fixedWeather2014' RDD to the 'fixedWeather2013' RDD to create a new RDD.  Then, use another Spark transformation to filter out days before September 21, 2013 and days after April 10, 2014."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "730\n",
      "202\n",
      "[('21/09/2013', ['2013', '09', '21', '�', '24.0', '', '5.5', '', '14.8', '', '3.2', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '', '', '21', '', '63', '']), ('22/09/2013', ['2013', '09', '22', '�', '20.1', '', '3.8', '', '12.0', '', '6.0', '', '0.0', '', '0.0', 'T', '0.0', '', '0.0', 'T', '', '', '26', '', '39', '']), ('23/09/2013', ['2013', '09', '23', '�', '17.6', '', '5.3', '', '11.5', '', '6.5', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '', '', '27', '', '59', '']), ('24/09/2013', ['2013', '09', '24', '�', '14.4', '', '3.9', '', '9.2', '', '8.8', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '', '', '27', '', '37', '']), ('25/09/2013', ['2013', '09', '25', '�', '11.3', '', '1.6', '', '6.5', '', '11.5', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '', '', '34', '', '50', ''])]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "import datetime\n",
    "def filterDates(weatherRDDRecord):\n",
    "    \"\"\"Skips records before September 21, 2013 and after April 10, 2014\n",
    "    Args:\n",
    "        weatherRDDRecord : key-value record where key is date and value is comma separated string of other weather fields\n",
    "\n",
    "    Returns:\n",
    "        True if key is in the desired period; False otherwise\n",
    "    \"\"\"\n",
    "    #converting date string to date format and then comparing\n",
    "    weather_date = datetime.datetime.strptime(weatherRDDRecord[0], '%d/%m/%Y')\n",
    "    if (weather_date <= datetime.datetime(2014, 4, 10) and weather_date >= datetime.datetime(2013, 9, 21)):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "aggregateWeatherRDD = fixedWeather2013RDD.union(fixedWeather2014RDD)\n",
    "print(aggregateWeatherRDD.count())\n",
    "desiredWeatherRDD = aggregateWeatherRDD.filter(filterDates)\n",
    "print(desiredWeatherRDD.count())\n",
    "print(desiredWeatherRDD.take(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1e) Merging commute time and weather data**\n",
    "#### We have to merge the commute time RDD 'deerfootPairRDD' and the weather RDD 'desiredWeatherRDD'.  Use an appropriate Spark transformation that uses the common key date key in both RDDs.  Print the number of elements and the first 5 elements of the resulting RDD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "202\n",
      "[('30/09/2013', (('Monday', '44', '7'), ['2013', '09', '30', '�', '14.3', '', '2.2', '', '8.3', '', '9.7', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '', '', '26', '', '72', ''])), ('11/11/2013', (('Monday', '38', '0'), ['2013', '11', '11', '�', '1.6', '', '-13.8', '', '-6.1', '', '24.1', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '3', '', '', '', '<31', ''])), ('05/12/2013', (('Thursday', '45', '8'), ['2013', '12', '05', '�', '-19.8', '', '-26.2', '', '-23.0', '', '41.0', '', '0.0', '', '0.0', '', '0.4', '', '0.2', '', '13', '', '33', '', '33', ''])), ('25/12/2013', (('Wednesday', '35', '0'), ['2013', '12', '25', '�', '6.5', '', '-10.1', '', '-1.8', '', '19.8', '', '0.0', '', '0.0', '', '0.0', '', '0.0', '', '12', '', '26', '', '63', ''])), ('07/01/2014', (('Tuesday', '45', '6'), ['2014', '01', '07', '�', '-3.0', '', '-9.9', '', '-6.5', '', '24.5', '', '0.0', '', '0.0', '', '0.0', 'T', '0.0', 'T', '8', '', '18', '', '32', '']))]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "combinedPairRDD = deerfootPairRDD.join(desiredWeatherRDD)\n",
    "print(combinedPairRDD.count())\n",
    "print(combinedPairRDD.take(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1f) Filter out weekends and unwanted weather fields to produce fina data for regression**\n",
    "#### Predicting commute times is more crucial for weekdays than for weekends, which have very light traffic.  So, we will exclude saturdays and sundays from the combined dataset.  We will also only consider using the number of accidents and the \"Snow on the ground\" fields of the combined dataset while building our predictive model.  While other fields, e.g., temperature, might also impact commute times, we will ignore them for the sake of simplicity.  As a result, we need to filter out the unwanted fields.  Use appropriate Spark transformations to produce an RDD whose each record has the format \"commute time, number of accidents, snow on the ground\". If the \"snow on the ground\" field is \"\", i.e., data was not logged that day, then assume that it is \"0\", i.e., assume that day had 0 snow on the ground."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "144\n",
      "[('44', '7', '0'), ('38', '0', '3'), ('45', '8', '13'), ('35', '0', '12'), ('45', '6', '8')]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def filterWeekends(combinedPairRDDRecord):\n",
    "    \"\"\"Skips records corresponding to Saturdays and Sundays\n",
    "    Args:\n",
    "        combinedPairRDDRecord : key-value record where key is date and value is a tuple containing comma-separated string values from deerfootPairRDD and desiredWeatherRDD \n",
    "    Returns:\n",
    "        True if key is in the desired period; False otherwise\n",
    "    \"\"\"                          \n",
    "    day=combinedPairRDDRecord[1][0][0]\n",
    "    if  day=='Saturday' or day=='Sunday':\n",
    "        return False\n",
    "    else:\n",
    "        return True\n",
    "    \n",
    "def extractRegressionData(combinedPairRDDWeekdaysRecord):\n",
    "    \"\"\"Outputs a string record with the format \"commute time,number of accidents, snow on ground\"\n",
    "    Args:\n",
    "        combinedPairRDDWeekdaysRecord: key-value record where key is date and value is a tuple containing comma-separated string values from deerfootPairRDD and desiredWeatherRDD \n",
    "    Returns:\n",
    "        record with the format \"commute time,number of accidents, snow on ground\n",
    "   \"\"\"\n",
    "    key = combinedPairRDDWeekdaysRecord[0]\n",
    "    commute_time = combinedPairRDDWeekdaysRecord[1][0][1]\n",
    "    number_of_accident = combinedPairRDDWeekdaysRecord[1][0][2]\n",
    "    Snow_on_Grnd_cm = combinedPairRDDWeekdaysRecord[1][1][20]\n",
    "    if  Snow_on_Grnd_cm==\"\" or Snow_on_Grnd_cm==\"''\" or Snow_on_Grnd_cm==\" \":\n",
    "        Snow_on_Grnd_cm='0'\n",
    "    return commute_time,number_of_accident,Snow_on_Grnd_cm\n",
    "\n",
    "combinedPairRDDWeekdays = combinedPairRDD.filter(filterWeekends)\n",
    "regressionDataRDD = combinedPairRDDWeekdays.map(extractRegressionData)\n",
    "print (regressionDataRDD.count())\n",
    "print (regressionDataRDD.take(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1g) Using `LabeledPoint` **\n",
    "#### We are now ready to use MLlib to do regression!  In MLlib, labeled training instances are stored using the [LabeledPoint](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LabeledPoint) object.  Write the parsePoint function that takes as input a raw data point in 'regressionDataRDD', parses it using Python's [unicode.split](https://docs.python.org/2/library/string.html#string.split) method, and returns a `LabeledPoint`.  Use this function to parse samplePoints (from the previous question).  Then print out the features and label for the first training point, using the `LabeledPoint.features` and `LabeledPoint.label` attributes. Finally, calculate the number features for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.mllib.regression import LabeledPoint\n",
    "import numpy as np,numpy\n",
    "# Here is a sample raw data point:\n",
    "# u'49,14,16'\n",
    "# In this raw data point, 49 is the label, and the remaining values are features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7.0,0.0] 44.0\n",
      "2\n",
      "[LabeledPoint(44.0, [7.0,0.0]), LabeledPoint(38.0, [0.0,3.0]), LabeledPoint(45.0, [8.0,13.0]), LabeledPoint(35.0, [0.0,12.0]), LabeledPoint(45.0, [6.0,8.0])]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def parsePoint(line):\n",
    "    \"\"\"Converts a comma separated unicode string into a `LabeledPoint`.\n",
    "\n",
    "    Args:\n",
    "        line (unicode): Comma separated unicode string where the first element is the label and the\n",
    "            remaining elements are features.\n",
    "\n",
    "    Returns:\n",
    "        LabeledPoint: The line is converted into a `LabeledPoint`, which consists of a label and\n",
    "            features.\n",
    "    \"\"\"\n",
    "    return LabeledPoint(line[0], [line[1], line[2]])\n",
    "\n",
    "parsedSamplePoints = regressionDataRDD.map(parsePoint)\n",
    "firstPoint = parsedSamplePoints.take(1)\n",
    "firstPointFeatures = firstPoint[0].features\n",
    "firstPointLabel = firstPoint[0].label\n",
    "print(firstPointFeatures, firstPointLabel)\n",
    "# d contains the number of features\n",
    "d = len(firstPointFeatures)\n",
    "print(d)\n",
    "print(parsedSamplePoints.take(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('44', '7', '0'),\n",
       " ('38', '0', '3'),\n",
       " ('45', '8', '13'),\n",
       " ('35', '0', '12'),\n",
       " ('45', '6', '8'),\n",
       " ('45', '3', '0'),\n",
       " ('44', '14', '5'),\n",
       " ('39', '5', '0'),\n",
       " ('58', '8', '1'),\n",
       " ('46', '5', '1'),\n",
       " ('48', '6', '1'),\n",
       " ('40', '4', '0'),\n",
       " ('45', '8', '0'),\n",
       " ('46', '2', '0'),\n",
       " ('43', '6', '3'),\n",
       " ('41', '1', '3'),\n",
       " ('56', '8', '0'),\n",
       " ('43', '1', '0'),\n",
       " ('48', '5', '0'),\n",
       " ('43', '5', '1'),\n",
       " ('45', '4', '12'),\n",
       " ('39', '9', '18'),\n",
       " ('36', '2', '4'),\n",
       " ('47', '7', '7'),\n",
       " ('42', '8', '3'),\n",
       " ('45', '5', '4'),\n",
       " ('37', '2', '9'),\n",
       " ('40', '7', '4'),\n",
       " ('42', '6', '0'),\n",
       " ('41', '3', '0'),\n",
       " ('60', '5', '21'),\n",
       " ('43', '9', '18'),\n",
       " ('37', '2', '0'),\n",
       " ('41', '2', '1'),\n",
       " ('35', '7', '0'),\n",
       " ('45', '4', '0'),\n",
       " ('44', '6', '0'),\n",
       " ('36', '1', '7'),\n",
       " ('43', '4', '5'),\n",
       " ('44', '7', '1'),\n",
       " ('40', '3', '4'),\n",
       " ('46', '6', '4'),\n",
       " ('41', '6', '2'),\n",
       " ('54', '5', '8'),\n",
       " ('43', '9', '11'),\n",
       " ('39', '8', '4'),\n",
       " ('46', '3', '0'),\n",
       " ('43', '7', '0'),\n",
       " ('46', '2', '2'),\n",
       " ('42', '2', '4'),\n",
       " ('41', '8', '1'),\n",
       " ('35', '0', '0'),\n",
       " ('77', '6', '4'),\n",
       " ('51', '4', '19'),\n",
       " ('52', '4', '17'),\n",
       " ('40', '5', '1'),\n",
       " ('46', '5', '0'),\n",
       " ('44', '6', '4'),\n",
       " ('41', '2', '0'),\n",
       " ('57', '9', '21'),\n",
       " ('45', '3', '2'),\n",
       " ('39', '6', '0'),\n",
       " ('42', '4', '5'),\n",
       " ('38', '2', '2'),\n",
       " ('39', '2', '1'),\n",
       " ('39', '5', '0'),\n",
       " ('38', '7', '10'),\n",
       " ('40', '4', '0'),\n",
       " ('47', '5', '0'),\n",
       " ('38', '2', '0'),\n",
       " ('42', '5', '4'),\n",
       " ('35', '3', '10'),\n",
       " ('46', '5', '0'),\n",
       " ('46', '6', '3'),\n",
       " ('51', '7', '3'),\n",
       " ('44', '9', '15'),\n",
       " ('47', '3', '0'),\n",
       " ('51', '4', '2'),\n",
       " ('49', '14', '16'),\n",
       " ('36', '2', '15'),\n",
       " ('38', '9', '8'),\n",
       " ('47', '7', '8'),\n",
       " ('45', '5', '3'),\n",
       " ('41', '8', '8'),\n",
       " ('34', '0', '0'),\n",
       " ('44', '4', '0'),\n",
       " ('44', '5', '4'),\n",
       " ('70', '15', '9'),\n",
       " ('43', '4', '0'),\n",
       " ('40', '5', '0'),\n",
       " ('40', '8', '4'),\n",
       " ('39', '5', '1'),\n",
       " ('42', '5', '1'),\n",
       " ('55', '7', '0'),\n",
       " ('48', '10', '2'),\n",
       " ('49', '3', '3'),\n",
       " ('35', '0', '9'),\n",
       " ('46', '3', '0'),\n",
       " ('87', '5', '14'),\n",
       " ('38', '5', '3'),\n",
       " ('43', '9', '1'),\n",
       " ('46', '2', '0'),\n",
       " ('46', '13', '0'),\n",
       " ('43', '3', '0'),\n",
       " ('44', '5', '11'),\n",
       " ('42', '5', '1'),\n",
       " ('97', '13', '21'),\n",
       " ('35', '3', '4'),\n",
       " ('45', '2', '1'),\n",
       " ('41', '4', '0'),\n",
       " ('52', '12', '2'),\n",
       " ('47', '3', '0'),\n",
       " ('46', '6', '0'),\n",
       " ('44', '2', '0'),\n",
       " ('78', '7', '0'),\n",
       " ('38', '3', '0'),\n",
       " ('40', '3', '0'),\n",
       " ('48', '6', '0'),\n",
       " ('47', '4', '0'),\n",
       " ('58', '6', '1'),\n",
       " ('49', '5', '2'),\n",
       " ('40', '3', '0'),\n",
       " ('38', '1', '0'),\n",
       " ('45', '5', '0'),\n",
       " ('45', '8', '0'),\n",
       " ('42', '3', '0'),\n",
       " ('43', '7', '0'),\n",
       " ('37', '6', '0'),\n",
       " ('38', '6', '2'),\n",
       " ('39', '5', '2'),\n",
       " ('39', '5', '2'),\n",
       " ('46', '11', '3'),\n",
       " ('44', '6', '0'),\n",
       " ('49', '1', '22'),\n",
       " ('39', '9', '8'),\n",
       " ('40', '0', '0'),\n",
       " ('35', '3', '8'),\n",
       " ('45', '4', '15'),\n",
       " ('45', '4', '10'),\n",
       " ('42', '6', '1'),\n",
       " ('40', '6', '0'),\n",
       " ('41', '2', '0'),\n",
       " ('41', '7', '1'),\n",
       " ('37', '8', '3')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regressionDataRDD.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(1h) Feature scaling **\n",
    "#### In learning problems, it is often necessary to \"scale\" features so that all features span nearly the same range.  One way to do this is to do (featureValue-meanOfFeatureValues)/standardDeviationOfFeatureValues.  For a given feature, the mean and standard deviation of its feature values are calculated.  Then, the mean is subtracted from each value and the result is then divided by the standard deviation. \n",
    "\n",
    "#### We will now do feature scaling for our dataset.  First implement a 'getNormalizedRDD' function that takes the RDD with non-scaled values and returns an RDD containing scaled features.  This function will first calculate the mean and standard deviations of the features in the non-scaled RDD.  It will then use a Spark transformation to scale each element of the non-scaled RDD.  This transformation should use the 'normalizeFeatures' closure listed below. Note that your implementation should generalize to any arbitrary number of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LabeledPoint(44.0, [0.6197334256477081,-0.7071495192910007]), LabeledPoint(38.0, [-1.7555186391921773,-0.15869603215686706]), LabeledPoint(45.0, [0.9590551491962631,1.6694822582902449]), LabeledPoint(35.0, [-1.7555186391921773,1.4866644292455338]), LabeledPoint(45.0, [0.28041170209915295,0.755393113066689])]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "import statistics\n",
    "import math\n",
    "\n",
    "feature_mean = []\n",
    "feature_standard = []\n",
    "\n",
    "def normalizeFeatures(lp):\n",
    "    \"\"\"Normalizes features in the LabeledPoint object lp.\n",
    "    Args:\n",
    "        lp - LabeledPoint object\n",
    "    Returns:\n",
    "        LabeledPoint: The object contains the label and the normalized features\n",
    "    \"\"\"\n",
    "    feature_List = []\n",
    "    number_of_features=len(lp.features)\n",
    "    for i in range(number_of_features):\n",
    "        normalized_feature= (lp.features[i] - feature_mean[i])/feature_standard[i]\n",
    "        feature_List.append(normalized_feature)\n",
    "    target = LabeledPoint(lp.label, feature_List)\n",
    "    return target\n",
    "\n",
    "def getNormalizedRDD(nonNormalizedRDD):\n",
    "    \"\"\"Normalizes the features of the LabeldPoints contained in nonNormalizedRDD.\n",
    "    Args:\n",
    "        nonNormalizedRDD - RDD containing non-normalized features\n",
    "    Returns:\n",
    "        returnRDD: RDD containing normalized features\n",
    "    \"\"\"\n",
    "    number_of_features=len(nonNormalizedRDD.take(1)[0].features)\n",
    "    for i in range(number_of_features):\n",
    "        feature_mean.append(nonNormalizedRDD.map(lambda x: x.features[i]).mean())\n",
    "        feature_standard.append(np.std(nonNormalizedRDD.map(lambda x: x.features[i]).collect()))\n",
    "    return nonNormalizedRDD.map(normalizeFeatures)\n",
    "\n",
    "normalizedSamplePoints = getNormalizedRDD(parsedSamplePoints)\n",
    "print(normalizedSamplePoints.take(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1i) Training and validation sets **\n",
    "#### We're almost done parsing our dataset, and our final task involves split it into a training set and a validation set. Use the [randomSplit method](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.randomSplit) with the specified weights and seed to create RDDs storing each of these datasets. Next, cache each of these RDDs, as we will be accessing them multiple times in the remainder of this lab. Finally, compute the size of each dataset and verify that the sum of their sizes equals the value computed in Part (1f)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "111 33 144\n",
      "144\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "weights = [.8, .2]\n",
    "seed = 40\n",
    "parsedTrainData, parsedValData = normalizedSamplePoints.randomSplit(seed = seed, weights = weights)\n",
    "parsedTrainData.cache()\n",
    "parsedValData.cache()\n",
    "nTrain = parsedTrainData.count()\n",
    "nVal = parsedValData.count()\n",
    "\n",
    "print(nTrain, nVal, nTrain + nVal)\n",
    "print(normalizedSamplePoints.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 2: Create and evaluate a baseline model **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(2a) Average label **\n",
    "#### A very simple yet natural baseline model is one where we always make the same prediction independent of the given data point, using the average label in the training set as the constant prediction value.  Compute this value, which is the average commute time for the training set.  Use an appropriate method in the [RDD API](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44.45045045045045\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "averageCommuteTime = parsedTrainData.map(lambda x: x.label).mean()\n",
    "                   \n",
    "print(averageCommuteTime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(2b) Root mean squared error **\n",
    "#### We naturally would like to see how well this naive baseline performs.  We will use root mean squared error ([RMSE](http://en.wikipedia.org/wiki/Root-mean-square_deviation)) for evaluation purposes.  Implement a function to compute RMSE given an RDD of (label, prediction) tuples, and test out this function on an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2909944487358056\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "import math\n",
    "def squaredError(label, prediction):\n",
    "    \"\"\"Calculates the the squared error for a single prediction.\n",
    "\n",
    "    Args:\n",
    "        label (float): The correct value for this observation.\n",
    "        prediction (float): The predicted value for this observation.\n",
    "\n",
    "    Returns:\n",
    "        float: The difference between the `label` and `prediction` squared.\n",
    "    \"\"\"\n",
    "    return (label - prediction)**2\n",
    "\n",
    "def calcRMSE(labelsAndPreds):\n",
    "    \"\"\"Calculates the root mean squared error for an `RDD` of (label, prediction) tuples.\n",
    "\n",
    "    Args:\n",
    "        labelsAndPred (RDD of (float, float)): An `RDD` consisting of (label, prediction) tuples.\n",
    "\n",
    "    Returns:\n",
    "        float: The square root of the mean of the squared errors.\n",
    "    \"\"\"\n",
    "    rmse = labelsAndPreds.map(lambda x: squaredError(x[0], x[1])).mean()\n",
    "    return float(math.sqrt(rmse))\n",
    "\n",
    "labelsAndPreds = sc.parallelize([(3., 1.), (1., 2.), (2., 2.)])\n",
    "exampleRMSE = calcRMSE(labelsAndPreds)\n",
    "print(exampleRMSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(2c) Training validation RMSE **\n",
    "#### Now let's calculate the training and validation RMSE of our baseline model. To do this, first create RDDs of (label, prediction) tuples for each dataset, and then call calcRMSE. Note that each RMSE can be interpreted as the average prediction error for the given dataset (in terms of commute time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Train RMSE = 8.515\n",
      "Baseline Validation RMSE = 9.806\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "labelsAndPredsTrain = parsedTrainData.map(lambda x: (x.label, averageCommuteTime))\n",
    "rmseTrainBase = calcRMSE(labelsAndPredsTrain)\n",
    "labelsAndPredsVal = parsedValData.map(lambda x: (x.label, averageCommuteTime))\n",
    "rmseValBase = calcRMSE(labelsAndPredsVal)\n",
    "print('Baseline Train RMSE = {0:.3f}'.format(rmseTrainBase))\n",
    "print('Baseline Validation RMSE = {0:.3f}'.format(rmseValBase))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 3: Train linear regression models using MLlib  **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(3a) `LinearRegressionWithSGD` **\n",
    "#### MLlib's [LinearRegressionWithSGD](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LinearRegressionWithSGD) implements linear regression with advanced functionality, such as stochastic gradient approximation and allowing both L1 or L2 regularization.  First use LinearRegressionWithSGD to train a model with L2 regularization and with an intercept.  This method returns a [LinearRegressionModel](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LinearRegressionModel).  Next, use the model's [weights](http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LinearRegressionModel.weights) and [intercept](http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LinearRegressionModel.intercept) attributes to print out the model's parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.mllib.linalg import DenseVector\n",
    "from pyspark.mllib.regression import LinearRegressionWithSGD\n",
    "# Values to use when training the linear regression model\n",
    "numIters = 500  # iterations\n",
    "alpha = 1.0  # step\n",
    "miniBatchFrac = 1.0  # miniBatchFraction\n",
    "reg = 1e-1  # regParam\n",
    "regType = 'l2'  # regType\n",
    "useIntercept = True  # intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.5032872987619497,1.4240757985103474] 40.49749271379786\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "firstModel = LinearRegressionWithSGD.train(parsedTrainData, numIters, alpha, miniBatchFrac, None, reg, regType, useIntercept)\n",
    "# weightsLR1 stores the model weights; interceptLR1 stores the model intercept\n",
    "weightsLR1 =  firstModel.weights\n",
    "interceptLR1 =  firstModel.intercept\n",
    "print(weightsLR1, interceptLR1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **(3b) Predict**\n",
    "#### Now use the [LinearRegressionModel.predict()](http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.regression.LinearRegressionModel.predict) method to make a prediction on a sample point.  Pass the `features` from a `LabeledPoint` into the `predict()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41.041829010489465, 35.876930022854005, 45.27575256766665, 38.220048035762986, 42.27518081676655, 48.28949956515512, 39.342989488983235, 42.15159521712135, 40.452755695615124, 38.49356972823013, 41.89124877124257, 36.794730206723905, 40.97344858737267, 36.72634978360712, 41.89124877124257, 35.94531044597079, 39.342989488983235, 39.60333593486202, 42.86425413164089, 42.6722881088789, 40.38437527249834, 39.13784821963288, 42.08321479400456, 40.19240924973635, 37.64414996747701, 44.81026485243752, 47.42690455781364, 36.794730206723905, 37.05507665260268, 41.041829010489465, 38.49356972823013, 40.19240924973635, 37.76773556712222, 39.795301957624005, 41.30217545636824, 41.233795033251454, 41.42576105601344, 42.932634554757676, 37.64414996747701, 37.31542309848145, 35.09589068521768, 41.233795033251454, 43.44015219992686, 42.91945930816931, 39.60333593486202, 39.342989488983235, 41.233795033251454, 36.794730206723905, 48.207943895449965, 38.16484285923457, 40.19240924973635, 39.795301957624005, 37.31542309848145, 37.05507665260268, 39.342989488983235, 43.64529346927721, 40.24761442626477, 39.342989488983235, 40.97344858737267, 41.82286834812579, 46.64586522017731, 37.64414996747701, 40.69992689490553, 44.82344009902589, 43.124600577519665, 43.97402033827277, 35.09589068521768, 38.49356972823013, 40.38437527249834, 50.18030510942333, 38.49356972823013, 42.932634554757676, 39.60333593486202, 41.041829010489465, 38.42518930511334, 37.439008698126656, 42.987839731286094, 40.124028826619565, 43.00101497787446, 46.138347575008126, 42.206800393649765, 38.685535750992116, 38.49356972823013, 45.809620706012566, 37.64414996747701, 40.19240924973635, 41.041829010489465, 37.64414996747701, 37.64414996747701, 40.19240924973635, 38.49356972823013, 40.452755695615124, 39.86368238074079, 37.64414996747701, 35.94531044597079, 39.342989488983235, 41.89124877124257, 41.041829010489465, 40.19240924973635, 40.7131021414939, 39.86368238074079, 39.86368238074079, 41.67293225530385, 44.82344009902589, 35.09589068521768, 39.72692153450722, 42.39876641641175, 40.452755695615124, 40.19240924973635, 36.794730206723905, 41.30217545636824]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "samplePoint = parsedTrainData.map(lambda x: x.features)\n",
    "samplePrediction = firstModel.predict(samplePoint)\n",
    "print(samplePrediction.collect())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3c) Evaluate RMSE **\n",
    "#### Next evaluate the accuracy of this model on the validation set.  Use the `predict()` method to create a `labelsAndPreds` RDD, and then use the `calcRMSE()` function from Part (2b)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.806415123335489\n",
      "9.400895139694786\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "labelsAndPreds = parsedValData.map(lambda x: (x.label, firstModel.predict(x.features)))\n",
    "rmseValLR1 = calcRMSE(labelsAndPreds)\n",
    "\n",
    "print(rmseValBase)\n",
    "print(rmseValLR1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3d) Trying a 2nd order model **\n",
    "#### The linear model is already outperforming the baseline on the validation set by more than a minute on average, but let's see if we can do better. Let's see if a 2nd order model will lead to better predictions. A 2nd order model will have as features [num of accidents, snoOnGround, (num of accidents)^2, (snoOnGround)^2,(num of accidents * snoOnGround)].  First, we will use a Spark transformation to convert the 'parsedSamplePoints' RDD to an RDD that has LabeledPoint objects for the 2nd order modelling. Next, we will scale the features of 2nd order RDD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LabeledPoint(44.0, [7.0,0.0,49.0,0.0,0.0]), LabeledPoint(38.0, [0.0,3.0,0.0,9.0,0.0])]\n",
      "[LabeledPoint(44.0, [0.6197334256477081,-0.7071495192910007,0.34345432596944103,-0.45583651643180145,-0.5334190884768721]), LabeledPoint(38.0, [-1.7555186391921773,-0.15869603215686706,-0.8986849482696025,-0.36442941055285755,-0.5334190884768721])]\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "def transformOrderTwo(lp):\n",
    "    \"\"\"Transforms the features in the LabeledPoint object lp into higher order features.\n",
    "\n",
    "    Args:\n",
    "        lp - LabeledPoint object \n",
    "\n",
    "    Returns:\n",
    "        LabeledPoint: The object contains the label and the higher order features\n",
    "    \"\"\"\n",
    "    accident_accident = lp.features[0]*lp.features[0]\n",
    "    snow_snow = lp.features[1]*lp.features[1]\n",
    "    accident_snow = lp.features[0]*lp.features[1]\n",
    "    return LabeledPoint(lp.label, [lp.features[0], lp.features[1], accident_accident, snow_snow, accident_snow])\n",
    "\n",
    "orderTwoParsedSamplePoints = parsedSamplePoints.map(transformOrderTwo)\n",
    "feature_mean = []\n",
    "feature_standard =[]\n",
    "print(orderTwoParsedSamplePoints.take(2))\n",
    "normalizedOrderTwoSamplePoints = getNormalizedRDD(orderTwoParsedSamplePoints)\n",
    "print(normalizedOrderTwoSamplePoints.take(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3e) Creating training and validation sets for 2nd order model **\n",
    "#### Follow the procedure in 1(i) to create a training and validation set for the 2nd order model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "111 33 144\n",
      "144\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "weights = [.8, .2]\n",
    "seed = 40\n",
    "parsedTrainDataOrderTwo, parsedValDataOrderTwo = normalizedOrderTwoSamplePoints.randomSplit(weights=weights, seed=seed)\n",
    "\n",
    "nTrain = parsedTrainDataOrderTwo.count()\n",
    "nVal = parsedValDataOrderTwo.count()\n",
    "\n",
    "print(nTrain, nVal, nTrain + nVal)\n",
    "print(normalizedOrderTwoSamplePoints.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3f) Train the 2nd order model using the training set **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Values to use when training the linear regression model\n",
    "numIters = 500  # iterations\n",
    "alpha = 1.0  # step\n",
    "miniBatchFrac = 1.0  # miniBatchFraction\n",
    "reg = 1e-1  # regParam\n",
    "regType = 'l2'  # regType\n",
    "useIntercept = True  # intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.2744834958982016,0.3738032705457207,0.5328339041179667,1.722457170470852,-0.6410533460273077] 40.44637521173379\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "secondModel = LinearRegressionWithSGD.train(parsedTrainDataOrderTwo, numIters, alpha, miniBatchFrac, None, reg, regType, useIntercept)\n",
    "# weightsLR1 stores the model weights; interceptLR1 stores the model intercept\n",
    "weightsLR2 = secondModel.weights\n",
    "interceptLR2 = secondModel.intercept\n",
    "print(weightsLR2, interceptLR2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3g) Evaluate RMSE of 2nd order model **\n",
    "#### This is similar to what you did in 3(c)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.806415123335489\n",
      "9.400895139694786\n",
      "9.272573007270356\n"
     ]
    }
   ],
   "source": [
    "# TODO: Replace <FILL IN> with appropriate code\n",
    "labelsAndPredsOrderTwo = parsedValDataOrderTwo.map(lambda x: (x.label, secondModel.predict(x.features)))\n",
    "rmseValLR2 = calcRMSE(labelsAndPredsOrderTwo) \n",
    "\n",
    "print(rmseValBase)\n",
    "print(rmseValLR1)\n",
    "print(rmseValLR2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** Conclusions**\n",
    "#### You would have noticed a slight improvement in accuracy with the 2nd order model from the 1st order model. To get further improvements we could try a few other things such as using an even higher level model, using more features from the commute time and weather data, and using more data.  We could also use more advaned regression techniques supported by MLlib such as ridge regression.  These are left as future exercises that you can try on your own.\n",
    "\n",
    "#### To finish things off, here is a plot that compares the actual commute times to those predicted by our best model.  This is basically a a color-coded scatter plot visualizing tuples storing i) the predicted value from this model and ii) true label.  Lighter colour predictions represent lower errors while the red ones represent large errors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABBoAAAJYCAYAAADMnIUCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3fmT1Hd+5/lXHpV1n0BxQ3FKQkISuiXUstRe93gUHkc4PNv7y/za83fNxkZMzHhjZsPenp4de2x161a3DmS6BYj7vgqqoO7KrMz9gRYjDJKg6isVgscjokMiM7/feme1CMhnfb6fb6nVarUCAAAAUIDycg8AAAAAPDiEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFCY6nIPQHL+/Pl8/vnnGR0dzfT0dH72s59lZGTk5vOtViuffPJJDh48mLm5uQwPD2fv3r0ZGhq6+Zq5ubm89957OXnyZJJk8+bN2bt3b9rb23/otwMAAMBDzIqG+0C9Xs+KFSuyd+/eOz7/+eefZ//+/dm7d2/+6q/+Kp2dnfnVr36V+fn5m6/5p3/6p1y5ciVvvvlm3nzzzVy5ciVvvfXWD/UWAAAAIInQcF/YtGlTnn/++WzZsuW251qtVvbv3589e/Zky5YtGRoayhtvvJFGo5EjR44kScbGxnL69On8yZ/8SVavXp3Vq1fntddey6lTpzI+Pv5Dvx0AAAAeYi6duM9NTExkZmYmGzZsuPlYpVLJ2rVrc/HixezatSsXL15MrVbL8PDwzdesXr06tVotFy9ezMDAwHd+nWazmXPnzqW3tzelUul7eS8AAABL0Wq1MjExkXXr1qVc9nPz+5XQcJ+bnp5OknR2dt7yeGdnZyYnJ5MkMzMztz3/1Wu+Ov67nDt3Lhs3blzitAAAAN+/06dP3/LDWO4vQsOPxGJWGbRarbs+rre3N0ny9ttv+w0LQCHGx8fz9ttv57XXXrur1XUA8F3OnDmT11577ebnF+5PQsN9rqurK8mNlQ1f/Xty6yqGzs7OzMzM3Hbs7OzsHVc63MlXQWLDhg133CsCAO7V6OhoOjs7s3HjxqxcuXK5xwHgAeJy7/ubi1ruc729vens7MyZM2duPrawsJDz589n9erVSW7sxzA/P59Lly7dfM2lS5cyPz9/8zUAAADwQ7Ci4T5Qr9dz7dq1m7++fv16RkdH09HRkZ6enuzevTv79u1Lf39/+vv789lnn6VarWb79u1JksHBwWzcuDFvv/12fvKTnyRJ3nnnnWzatMlSVQAAAH5QQsN94PLly/nlL39589cffvhhkmTnzp15/fXX89RTT6XRaOTdd9/N/Px8hoeH8+abb6ZWq9085o033sj777+fX/3qV0mSzZs3Z+/evT/sGwEAAOChJzTcB9atW5d//+///Tc+XyqV8txzz+W55577xtd0dHTkpz/96fcxHgAAANw1oQEAACjclStX8vsPf5frY+NZqNdTaWtL/4qhPPHicxkaGlru8YDvkdAAAAAUYmFhIUcPfZn9736UiQPH0zx4OuW5ekqtpFVKLnW05dj/fD99j23Nkz95KVt3bE+5bH96eNAIDQAAwJLNzc3l//vP/yWjf//bVM5dTanVSuVrz5daSXWmnnx6NNf3Hctv3v4sB/7VS/lX//tf3bL3GPDjJx8CAABLMjc3l//3//yPGf3P/5jq2SsptVrf+vpSs5Xq6dFc+k//kF/+X/8p9Xr9B5oU+CEIDQAAwKK1Wq38j//7/8m1v3sv1en5ezq2OjmXsb99N3//X/42re+IE8CPh9AAAAAs2tmzZ3Plnc9SnZpb1PHViZlcevezXLp0qeDJgOUiNAAAAIu279fvJccuLO0kX57NZ795r5iBgGUnNAAAAIsyOTmZqwePpdxoLuk85fpCLh84lpmZmYImA5aT0AAAACzK0QMHM7//WCHnmv38SE4cOVrIuYDlJTQAAACLMjl2LeX5hULOVZ6vZ2JsvJBzActLaAAAABalPj+fNAu6W0Szlfrcvd21Arg/CQ0AAMCidHR1pVUp5iNFq1JOR3dXIecClpfQAAAALMrK9WuTwZ5CzlVa0ZeVa1cXci5geQkNAADAooxs25qOp7cv+TytJJ1PbsvGzZuXPhSw7IQGAABgUarVajY/tSuNrtqSztPo7ci2Z59KuezjCTwI/E4GAAAW7cmXnk91z7YsdkvIVpLaMzvyxHN7ihwLWEZCAwAAsGh9fX3Z/Rf/W1o71i3q+NZjG7PnL/8s3d3dBU8GLBehAQAAWJI9Lz6fnT//V2k+uuGuVza0kjSf2JRd/8eb2f2M1QzwIKku9wAAAMCPW6lUyit/+kZ6+vuz/x/ezuynh1O9MpHSHV7bKiWNlX3peGZHnvnzN7Lr6ad+8HmB75fQAAAALFmpVMpTzz+bR598Iod+/4cc+PCTTB88kcal8aTeSNra0rZ6IF2PjmTXK89n567HUqstbRNJ4P4kNAAAAIVpb2/Pk88+k93P7Mno6GhmZmYyNzub9o6OdHV1ZcWKFSmV7rTWAXhQCA0AAEDhSqVSVq1atdxjAMvAZpAAAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAw1eUegO/WbDbzySef5MiRI5menk5XV1d27tyZZ555JqVSKUnSarXyySef5ODBg5mbm8vw8HD27t2boaGhZZ4eAACAh4nQ8COwb9++fPHFF3njjTcyODiYy5cv5ze/+U1qtVp2796dJPn888+zf//+vP766+nv78+nn36aX/3qV/n5z3+eWq22zO8AAACAh4VLJ34ELl26lJGRkWzatCm9vb3ZunVr1q9fn8uXLye5sZph//792bNnT7Zs2ZKhoaG88cYbaTQaOXLkyDJPDwAAwMNEaPgRWLNmTc6ePZvx8fEkyZUrV3Lx4sVs2rQpSTIxMZGZmZls2LDh5jGVSiVr167NxYsXl2VmAAAAHk4unfgReOqppzI/P5+/+Zu/SalUSqvVyvPPP5/t27cnSaanp5MknZ2dtxzX2dmZycnJe/pa4+PjGR0dLWZwAB5qY2Njt/wTAJbqqx++cn8TGn4Ejh49msOHD+enP/1phoaGMjo6mg8++CDd3d3ZuXPnzdd9tTHkUrz99tu3BQsAWIq33npruUcA4AExMzOz3CNwF4SGH4GPPvooTz/99M0VDENDQ5mcnMxnn32WnTt3pqurK0lu3pHiKzMzM/ccDV577bVs3LixuOEBeGiNjY3lrbfeurmZMQAs1enTp5d7BO6C0PAj0Gg0blut8PVf9/b2prOzM2fOnMnKlSuTJAsLCzl//nxeeOGFe/paAwMDN88BAEUYHBz0ZwsAhZiYmFjuEbgLQsOPwObNm/PZZ5+lp6cng4ODGR0dzf79+/PII48kuREddu/enX379qW/vz/9/f357LPPUq1Wb66CAAAAgB+C0PAj8Morr+Tjjz/Ou+++m5mZmXR1deWxxx7LM888c/M1Tz31VBqNRt59993Mz89neHg4b775Zmq12jJODgAAwMNGaPgRqNVqeeWVV/LKK69842tKpVKee+65PPfccz/gZAAAAHCr8nIPAAAAADw4hAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAwQgMAAABQGKEBAAAAKIzQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAAAAFEZoAAAAAAojNAAAAACFERoAAACAwggNAAAAQGGEBgAAAKAw1eUe4Mfkgw8+uOvXvvzyy9/jJAAAAHB/Ehruwejo6F29rlQqfc+TAAAAwP1JaLgH/+bf/JvlHgEAAADua/ZoAAAAAApjRcMSXLp0KceOHcvk5GSazeYtz/3sZz9bpqkAAABg+VjRsEhHjhzJ3/3d32V8fDwnTpxIs9nM2NhYzp07l1qtttzjAQAAwLIQGhZp3759efnll/Pnf/7nqVQqeeWVV/Lzn/88W7duTU9Pz3KPBwAAAMtCaFik69evZ9OmTUmScrmcRqORUqmU3bt358CBA8s8HQAAACwPoWGR2tvbU6/XkyTd3d25evVqkmR+fj6NRmM5RwMAAIBlYzPIRVqzZk3OnDmToaGhbN26Ne+//37OnTuXM2fOZP369cs9HgAAACwLoWGR9u7dm4WFhSTJnj17Ui6Xc+HChWzZsiXPPPPMMk8HAAAAy0NoWKSOjo6b/14qlfL0008v4zQAAABwfxAaFmlycvJbn3fnCQAAAB5GQsMi/cf/+B9TKpW+8flf/OIXP+A0AAAAcH8QGhbpr//6r2/5dbPZzOjoaPbv35/nn3++8K83NTWVjz76KKdPn06j0cjAwEBee+21rFq1KknSarXyySef5ODBg5mbm8vw8HD27t2boaGhwmcBAACAbyI0LNKKFStue2zVqlXp7u7O559/ni1bthT2tebm5vK3f/u3WbduXf71v/7X6ezszPXr19Pe3n7zNZ9//nn279+f119/Pf39/fn000/zq1/9Kj//+c9Tq9UKmwUAAAC+TXm5B3jQ9Pf35/Lly4Wec9++fenp6cnrr7+e4eHh9Pb2Zv369enr60tyYzXD/v37s2fPnmzZsiVDQ0N544030mg0cuTIkUJnAQAAgG9jRcMizc/P3/bY9PR0Pv744/T39xf6tU6ePJkNGzbkH/7hH3L+/Pl0d3dn165deeyxx5IkExMTmZmZyYYNG24eU6lUsnbt2ly8eDG7du0qdB4AAAD4JkLDIv2H//AfbtsMstVqpaenJ3/6p39a6NeamJjIgQMHsnv37uzZsyeXLl3K+++/n0qlkp07d2Z6ejpJ0tnZectxnZ2d33l3jH9pfHw8o6Ojhc0OwMNrbGzsln8CwFKNj48v9wjcBaFhkf7iL/7ill+XSqV0dnamr68v5XKxV6S0Wq2sWrUqL7zwQpJk5cqVGRsbyxdffJGdO3feMsNSvf3227cFCwBYirfeemu5RwDgATEzM7PcI3AXhIZFKpVKWb169W1Rodls5vz581m7dm1hX6urqysDAwO3PDY4OJjjx4/ffD65cenGV/+e3PhNeK/R4LXXXsvGjRuXODEA3FjJ8NZbb+WNN97I4ODgco8DwAPg9OnTyz0Cd0FoWKRf/vKX+Xf/7t/d9kF+fn4+v/zlL/OLX/yisK+1evXqXLt27ZbHxsfH09vbmyTp7e1NZ2dnzpw5k5UrVyZJFhYWcv78+ZurIO7WwMDAzXMAQBEGBwf92QJAISYmJpZ7BO6Cu04sUqvVuuPjs7OzqVaL7Te7d+/OxYsX89lnn+XatWs5cuRIDh48eHOTx1KplN27d2ffvn05fvx4rl69ml//+tepVqvZvn17obMAAADAt7Gi4R79/d//fZIbH+5//etfp1Kp3Hyu1Wrl6tWrWb16daFfc3h4OD/72c/y29/+Np9++ml6e3vz8ssvZ8eOHTdf89RTT6XRaOTdd9/N/Px8hoeH8+abb6ZWqxU6CwAAAHwboeEeffXBvdVqpa2t7ZbVC+VyOY8++ujN204WafPmzdm8efM3Pl8qlfLcc8/lueeeK/xrAwAAwN0SGu7R66+/nuTGvghPPfVU4ZdJAAAAwI+ZPRoWaceOHZmamrrt8WvXrtmgBAAAgIeW0LBIv/nNb3Lx4sXbHr906VJ+/etf//ADAQAAwH1AaFik0dHRO276ODw8nCtXrizDRAAAALD8hIYlqNfrtz02Pz//jbe+BAAAgAed0LBIa9euzb59+9JsNm8+1mw2s2/fvqxZs2YZJwMAAIDl45YJi/Tiiy/m7/7u7/I3f/M3N8PChQsXMj8/n7/4i79Y5ukAAABgeQgNizQ4OJh/+2//bf7whz/kypUrqVar2bFjR5544gl3nQAAAOChJTQsQXd3d1544YUkydzcXI4cOZL/9t/+W65cuZJf/OIXyzwdAAAA/PCEhiU6e/ZsDh06lOPHj6e3tzdbtmzJa6+9ttxjAQAAwLIQGhZhcnIyX375ZQ4dOpRGo5GtW7em2Wzmz/7szzI4OLjc4wEAAMCyERru0X//7/89Fy5cyKZNm7J3795s2LAh5XI5X3zxxXKPBgAAAMtOaLhHZ86cyRNPPJFdu3alv79/uccBAACA+4rQcI/+8i//MocOHcp//a//NQMDA9mxY0e2bdu23GMBAADAfUFouEerV6/O6tWr88orr+TIkSM5dOhQPvzwwyQ3Vjt0d3enVqst85QAAACwPISGRapWq3n00Ufz6KOPZnx8PAcPHsy+ffvy29/+NuvXr8+f//mfL/eIAAAA8IMTGgowMDCQl156KS+88EJOnTqVgwcPLvdIAAAAsCyEhgKVy+WMjIxkZGRkuUcBAACAZVFe7gEAAACAB4fQAAAAABRGaAAAAAAKIzQAAAAAhREaAAAAgMIIDQAAAEBhhAYAAACgMEIDAAAAUBihAQAAACiM0AAAAAAURmgAAAAACiM0AAAAAIURGgAAAIDCCA0AAABAYYQGAAAAoDBCAwAAAFAYoQEAAAAojNAAAA+JVquVubm51Ov1tFqt5R4HAHhAVZd7AADg+1Ov13PyxLGcPvVlKuWFtLWV0mol9Xoz5UpXtm1/ImvXrUupVFruUQGAB4TQAAAPoEajkX/+/ONcHz+fkY0dee3FwVQqty5knJ1r5NjJz3Lgi99l88hj2bZ9p+AAACyZSycA4AEzNzeXd9/++6xdcT2vv7I6Ixv7b4sMSdLRXs2unUP56d5VmZs6nE8/+dAlFQDAkgkNAPAAaTQaef+9f8xTj7Vn7eqeuzqmVCrl8UeGMtA9ls8+/eh7nhAAeNAJDQDwAPns0w+za1tbBgc67/nYbSMDqeZSTp08/j1MBgA8LIQGAHhAzM3NZXZ6NKuHuxd9jl07h3Ls6BcFTgUAPGyEBgB4QBw9+mW2brr3lQxfV62W09ezkKtXrxY0FQDwsBEaAOAB0Gq1cv7s0axbc3f7Mnyb7SO9Ofzl7wuYCgB4GAkNAPAAmJmZSW93uZDbU/b1tmd25loBUwEADyOhAQAeAPPz86nVlh4Z/he3uQQAFkdoAIAHQBErGQAAilBd7gEAgKWr1WqZm28Wd8KWcAE/RrOzszlz5lRmZ6fTaDTS1lZLb29/1q/fkEqlstzjAQ8JoQEAHgAdHR2Zmk6azVbK5aVFgmvXZ9PVM1jQZMD3rdVqZXT0co4cOZCFhals3jyQwcFaKpVy6vW5jI+fzG9+sz9DQ2uzffsj6elZ+qaxAN9GaACAB0CpVMq6Ddtz7sLpbFjXt6RzHT4+mR2PvlbQZMD3qdFo5MMP30lPz0KefHJVurvX3PaaVav6sn37cEZHJ7Jv39tZsWJTHn30cZdcAd8bezQAwANiy5btOXZqZknnqNcXMjVTzcDAQEFTAd+Xer2ed9/9x+zY0Z2nn96Y7u6Ob3xtqVTKqlV92bt3a5rNy/n8809+wEmBh43QAAAPiPb29nT3rs7ZC5OLPsf+A1ezbfsTBU4FfB+azWY+/PCdPP74UFav7r/r40qlUh5/fF2q1es5dOiL73FC4GEmNADAA+TpPS/myIlmRq9M3/OxB4+MpdqxIRs2bvoeJgOKdObM6axaVc6qVYu7VOrxx9flwoXjmZubK3gyAHs0AMADpVKp5OW9P82H77+Vkdnr2biu9zuvw15YaGb/gaspta3Nk0898wNNyr0aGxvLsaOHMjU1kVaaKaWU9vbObN68PavXrHG9/UPmxIkv89JL6xZ9fKlUyrZtQzl27HAee8wqJqBYQgMAPGBqtVpefe3P8vv9n+XI+6ezcW1bRjb2pa3t1lvbTU3P58jx67kyXsrW7Y9nZGTbMk3MN2k2mzl58kROnvgyvb3lbN++Kn19AymVSmm1WpmZmc+xYwfyxRefZt36Ldm2bUfa2tqWe2y+Z9euXUtHRyu12tL+Kr9+/WDeeuuojSGBwgkNAPAAKpfLefKpZ9Ns7snpUyfz/icHUko9pVIrrVYrzVYp7e192b7jpTz57CofMu5D9Xo9H37wToZXl/Pqq5tSrd4aikqlUrq62vPEE+vTbLZy9uyVvPP2P+TFl/4k3d3dyzQ1P4Tjxw9n+/YVSz5PqVTK8HBnLl++nOHh4QImA7hBaACAB1i5XM7mkS3ZPLIlyY2fkJdKJWHhPtdoNPLeu/+Uxx8fzKpV373RX7lcysaNKzM42JMPP3grL7/y03R1df0Ak7Icpqcn09u7qpBz9fa2Z2pqqpBzAXzFZpAA8BApl8siw4/A7377Xh59dOCuIsPX9fR05IUXNuTDD36TRqPxPU3Hcms06retcFmstrZyGo35Qs4F8BWhAQDgPjI2NpZabS5r1gws6vje3s5sHunJqVMnC56M+0WlUk2j0SzkXI3GQqpV+3oAxXLpBADwra5fv57DX36RiYkrKZdbSZJWq5RSqZqRkUeyYeOmlMt+dlGUw19+kUcfXdqy+M2bVuSddw9n61YbfD6Iurp6Mjk5m8HBpe/FMTFRz8qV9vQAiiU0AAB3dPHChRw69Hk62heyY/tgBgc33vL8/HwjJ04eya/f2p9Vwxuya9eTqVSKWc79sJqfn8/s7LX09a1c0nmq1Ur6+yu5evVqhoaGCpqO+8XIyPYcPfpxnntuaYGg1WrlwoWp7NplI0igWEIDAHCbw18ezOjokbz0wrpvvIVerVbNzh3D2bG9lTNnx/PuO/+Yl17+k7S3t//A0z44zp87l/Ubego518jmoZw6eUxoeAANDg5mcrKZen3httvW3osLF65lzZpN9m0BCmedIwBwi6NHvszE9eN56YWN3xgZvq5UKmXjhsHsfrw3H7z/lk0Il2B2biZdnbVCztXVVcvc3Ewh5+L+MzKyM8ePX1708a1WK0eOXMm2bTsLnArgBqEBALhpdHQ0Fy98mT1Pr7vnn3IODXXnsUd68uknH35P0z34FhYaqVSK+etZpVLOwsJCIefi/rNp0+acPz+fK1cmF3X8oUMXMzS0IR0dHQVPBiA0AMD3YnZ2NteuXcuVK1dy/fr1Rf2Uf25u7pZz1Ov172HSW315aH+efmr1HSNDs9lMo9FIvV5Po9FIs3n7rverV/elXh/PzIyfpC9Gra098/ViVoTMzy+krfbdqyOW478zlq5cLufll/8kn39+OaOjE/d07KFDFzMxUcuuXbu/p+mAh509GgCgIM1mM2d5yWOrAAAgAElEQVROn86JEwfTVq2ns7OaSqWUer2ZyclG2jsGsmPn4996zXyr1cq5s2dz7NiBlMtz6e5sS7X6x3NMNVKt9WT79sezcuXKwq+rnpmZSbM5la6uFV+bJ5mfn8vMzHSSZiqVUkqlUlqtVhYWWimVyuns6E5brZavxtm+tT9HjhzM2rUb7/yF+EZ9/QO5ePFMNqxf+rnGxibT13vnW2Q2m82cO3c2x44dTLXaSFdXWyqVSur1hUxOzqdW682OHbuyYsWKOx7P/aFWq+XVV3+aDz98OytWTGTbtpXp6PjmuDQ2NpWDBy+mp2dtnnvuKXszAN8boQEACnD06OGcPH4g69d15qXnh1Jru/2P2OsTszly9MP88+fJ00+/nIHBwVueP3XyRA4f3p+1azrywrNDaW+//RyTU3M5cvST/H5/M7t3P5+Vq5Z2G8Rb38OhbN/af/PXc3NzmZ6eTK2tnN6e6h2X9C8sNDMzO5mp6Va6u3pTa69leLgvfzhwKqtXF/Bp+SGzatWq/H7/TJrNVsrlpX0IPHZ8PC+++Oxtj584cSzHjh3IunU9efHFtWlvb7vtNRMTMzly5LP88z838tRTz2doSHC4X9VqtfzkJ3+aCxfO5+OPD6RSqWdkZDBdXbVUq+XU6wsZG5vOyZPX0tOzIrt2vZL+/v7vPjHAEggNALAErVYr+//5szQXLuaN1zZ8608I+3o78szTazM7W88Hv3s7u3a9kNVr1iZJDh74Qyaun8wbr2381g+YPd3tefrJtanXF/LBRx9k6/Y92bChmJUDV0Yv5PFHb8wzMz2d+fpMBvrbv/U9VSrl9HS3p9lq5fr1iSw0u9LZ2ZmhoVqmpqYKmethUiqVsn79lpw9eyUbNy7+FpcTEzNpr/XedgeQP/zh88zOXsobb2z71v9fe3s7s2fPxszN1fPBBx/kkUf2ZO1a4eh+VSqVsnbtuqxduy5TU1M5depELl6cTqPRSFtbLT09a7J374tpa7s9KgF8H4QGAFiCAwd+n0r5cp7cteauj+noaMtPXt6Qdz/8bdraXs3Y2JXMTJ3Kc8/c/QaMbW2VvPrKxrz34b7U2moZXr16sW/ha1oplUqZnZ1NvTGTvr6O3O3P1MulUvr7O3L9+nTK5XLaa2XX+i/Slq3b8967/yPr169Y9KqGAwcuZPuO52957NChA1lYGM2zz2666/O0t7flJz/Zlvfe25darT0rViw+fvDD6O7uzmOPPb7cYwAPOZtBAsAiXblyJdfHT2TXI/d++UK1Ws7LL6zL7373ds6eOZCnn1p7z9dLl8vlvPT8+uzf/2Fht5RcWGhmZmYyvT13Hxm+UkrS19ue6emJtHJjtQf3rlarZevWJ/LJpycX9T08fPhi2juGb9lfYXx8PFeunMju3fe+KqFSKeell0ayb9+Hd9wAFAD+JSsaAGCRjnz5hzz+2J03ZWw2W7l46XqmZ+bTqC+k2lZJV2ctq4f7bv6Uur1WzYrBVrq62m47R6vVyvx8Pc3mQlqtGysNKpXKjU0Xv/a6trZKtmzuzcmTx7Nt244lvqNSZmam/zjPN79qdraRS5cnMze3kGarlVpbJUNDnenv60ipVEpnZ1smrk9k1eo7L9NutVoZHR3NxMRE6vOzqVTa0tHZmTVr1qRa/e6/mrRarVy9ejXXr13LfH0ulXI17R0dWbNmzQOzNHzzyJbMzc3mdx+fzHPPbr7rlQ0HD17IxERbnnt+zy2PHz78RR5/fM2iN/+r1arZtKkvZ86cyqZNI4s6BwAPD6EBABZhbm4u8/PX0tuz4ZbHZ2bmc/T4pVy8dD1r13Snt6eW7q5K6vVGxsanc+DQ2axe1Z+tW4bT3lHNxvVd+fLItZvHLywsZGZmOvX6fGq1ciqVUsrl0h/DQzNTU63Uah3p7OxMuXxjYeKmjYN55/3DSw4NPb2DuXz5SkZGBm97rtVqZfTKdI4cvZLGQivr1/Wlq7uWcimZry/ky8NXMj09ny0jg1m/rj/nzl/NppHOW84xPz+fE8eP5cyZo1m5sj0DA+3p7KtmodnM9NR83nlnX/r7h7N9+6Pp6+u7bYZGo5ETx4/l9OmjGRxsy9BQR/o6bhw/M13Pe+99np6eldmx47EHYrO7nY88llMnu/LrX+/P5pH+bN60ItVq5bbXtVqtnD13NceOjWXF0MY89/zuW4JCvV7P9PRY+vuXtqHjyMiKvP/+l0IDAN9JaACARTh27HC2jvTe/HWr1coXB87l6vhkdmwbzOOPbb7jT48fe2RlLlyczMefHk1vT0d27uhLrS25PjmbUqueZnM+nZ1t6e6+w6UL7UkrydxcI9eujaW9vTNdXV2pVssZ6CvnypUrS7odYW/PQE6cPJot/yI0zM018tHvTqevvzO7n1iTnp72247dvGkw9fpCjp8Yyz/805GUSuVcGx+/+fyJE8dy/Ngfsm3bYF5/feRmJPm6nTuTK1cmcuCL91Ou9OfZZ1+8+bozZ07n0KF92bZ1MK+9NnLHO2Ds2LEm4+NT+fLQR1lY6MzzL7ySSuX2D+Y/Jps2b8669etz+tTJvPvu4fT2ltPf35G2tnIajWampuoZHZ3NunUjefHFZ2/b/DG58b3fsuX2eHSv2tqq6epKrl279kCEHAC+P0IDACzC2NVL2bb5RmhotVr5+LOT6e0p5yevfPsdIEqlUtau6c2a1T35wxcX8/sDo1mxoiPnzl7K5k196ezo+Pbjk3S0V9PeXs3U1GwmJ5vp6enJqlUdGbu6tNAwOzeVxkIyN99Ie+3GXxFmZup5/6NTeXbP+gwMdH7r8W1tlezcsTIrVnTl40/O5+y500mSE8ePJhnP669/+50OkmTFit6sWNGb06ev5r1338ore1/PiRPHcvny0bzx+vY7BoqvGxjozvPPd+f8+fG8884/5tVXf3pXl2Pcz6rVarZs3ZYtW7dlfHw8U1NTqdfrae+opn+gI7ufXPGt39erVy/nySeLCQPDw90ZG7sqNADwrWwGCQCL0GjU0/bHZey//+Js+nsreXTn3e/IXyqV8ugjK9PbU8vZ8xNptRbS2XH3+wuUcuNWl8l8pqenU6tVM1+fu8d3cav5+blsGRnOH764nCR/vIXm6Tz/3MbvjAxfaTSaqVQqee0nW3P92qUkyfT0xTz77MZ72h9g48ahbN3Wlbf+6X/k4sXDefGFO6+C+CZr1w5k12MD+eCDtx+oDQwHBgayfv36jIyMZMOGDVm58s57hHzd/PxcarViYkutVsn8/NL+OwPgwSc0AMBi/PHD3fXrM5mcms7OHYtbSfDIjqEsLDTTXFjcHRq6u9szPz+TxsJCyqWl/bFeLpczPNyfSrk9Xx6+kgMHL+exR1elr/f25fh3srDQzMTkfHp7O1OtlvP4ruEkyaOPLm4TwnVrB9LZNZONG/oWdfzwcH+Gh8s5eeL4PR/7ICmXy4XdAaTZbN1T8AHg4fTjXksIAMukva0js7ONHD56MY/+i8hwdWwmp05fy9xcI41GM221Snp7atmyeTAdHbf+0dtqtfLIjqEcPnoti1FK0tlZzYWLk6l1fvtlG9+lvdaRubnpPPnkpnzy6fGcOz+WJ3avuatj641mJv8YGSrlUhqNhYyNzSRJFpoLN1/XarUyOzuXen0+rdaNlQblcjltbe1p72i/ZV+KubnZPL5rOPv3X8rGjfd+C9Ek2bZ1OO++ezhbtm5b1PEPglqtPbOz9fT0LH2/itnZemq1b7+8BwCEBgBYhLXrNufEyYOZmJzJ4OBwFhaaOXX6Wk6eGk9vX3u2jAymq7MtlUopjUYzY+Oz+d2nZ1OtlLN921BWrexOqVROq9VId1ctzWYzjUYz1eq9/7S4Vqvm6LHRvPGndxcFvvk9bcyJY7/LqpW9WbWqL+0dybVrc6nVKunsqN52i8VWkvn5hczM1FMqldPf13XzNbOzjVy5OpUkmZudSWOhNzPT02k05tPeXklXVzXl8o0Pvs1mK3NzMxkfm0yt1pGOzs5UyuXMzk6nr68zbbVkYmImvb13d/nG11WrlfT2VXL16tUMDQ0t6fvzY7Vu3eacOnU4u3atXfK5zp2bzAsvrC5gKgAeZNa+AcAibNi4MYeOXMmmDX2Zmann7XdPZqHZyqt7N+fZPesyNNiZjo5q2toq6exsy7q1vfnJ3s3ZvXt1Tp2+lk8+O5dWq5l6o5lyuZTNm/pz5uziVjXMzNTTSumWuzwsxtDQUK5PNNNoLOTkqdE8unM4AwPdqVSquT4xn/Frs7k+MZfrE/O5dn0u4+OzaTRa6e3tSn9f583I0Gy1cnVsNv39Ny65qDfqmbg+nvb2ZGCgM11dtVSr5ZTLN27dWa2W091dy8BAZ6rVhVy/NpaZmdmUy0m5XMq2bSty7NiFRb+vHdtX5ujRg0v63vyYrVu3LhcuTC358omJiZm0tfXe8c4WAPB1QgMALEK5XE5nZ08q1eT9j07n2WfWZvu2oe9ckdDTXcuzz6zL0FBnPvr4bGZnGqlUyhnob8/k1OI22Tt67GpGNq/M5OTEoo7/us2bd+bkqatJq5m2tsrNu1wM9Helv6873d2d6epqT29vZwYGutPd1Z7Kv1jpMDPTSKNRSm9PLUlSKSf9/R2p1Sr5tq0WSqWkvb2a/v7OzM5O5saaiWSgvzNTU7OLfk+9vZ2ZmZ5c9PE/dqVSKWvWbMr580sLUUePjmb79scKmgqAB5nQAACL1NfXnz98cTnPP7sufX33dt36lpHBrFrZnYOHxzI/v5BKpZx6/d7vjnDt2myuXJ3P8Kq+1Jd414kk2bR5JCdOTqWxcPsspVJSKZdSrZRTLpVyp2bQWGhmfr6ZlMpp/TEUdHXd/d00kqRcTvr6OlJvNFKvL6RaLafRaCzm7fxx7tLNWR5W27btzIEDo5mfX9z38erVqVy/niXdPhWAh4fQAACLNDU1le3bV6azq+2el6U3W62MjAzcuB1ktZrx8Znc68r2icm5fLzvQl56cUcWms1Uqvf2gf5OKpVKXnjxTzI+PpuF5r0NtNBsZWJiPn19A6mWS5mdrf/xmXu/Y0SpVEpfb0emp+cKudPBnbPIw6O9vT1PP/1S3n//eOr1e4sN169P57PPLuTFF3+yqLt/APDwsRkkACxCq9XK9MxENqxfk0qlmsbCjUsgyt/xQayVpNlsptlMyuVKtm8byqnT4+npbs+JU9eyZWQwvT3tOXl6PGfPXk9KSat1YzVBuVTKyOaBrF3Tm0uXp3Lg0NW8/OLOdHS0ZXq6ns6OrkLeW29vb3r7VmR8fDa9vbXU2r79bgWtJPX6Qqam6untG0ilUs7c/ELm5hb++HxrUR9QbxzTysTEXDo6arc9Pzs7n2PHLubS5fGUSzfmKCWp1dqybdvarFx547aYs7PzqbY9HPsK1Ov1nDx5POfOnfjjXT1ufO/L5Vq2bHkkjz/+Qt5557d59tl16e/v/tZztVqtnDs3li+/vJZXXnnD3gwA3DWhAQAWYXR0NBvW96deb6Wrs5xU2tJsLmShtfDHTQ7Lt/wMvdVqpdlspdlqpVyqpFotp9VKVq7syoGDlzM2Pp8XX9iRdz84nkajkcceHc4rr2xOtfq/PuTPztZz8NBoPvjd2fT1dOYnrz6Srq4bH/5On5nK3lfXFfb+Nm7Ymqnpq6nVSpmamk1HRyXt7W35+nYMzdaNmebmFlJtq6W/f+jmhpBjY5O5fv2rfRVK37o3w52UyqW0Flrp6GjLp5+ezfYdm24+NzY2mYOHzmSh0cjWbSvy2GPbbgkZU1NzOXLkUn7/+5PZuHFV5uZb2bx552K/FT8KU1NTOXDgnzM1dTUjIwN59dWNt6wCmZur5+jRw7lwYSoDA6tz4MB05ubOZ8uWwWzYMHjLa+fnGzl+fDRnz05keHhjXn31ubS1LX21DAAPD6EBABbh5InD2bljKK3m3I3LFsrllCvVtNK6eavKf6lcrqRa+V8BolRKFlLKypXdOXV6MgcPnc0rL23O4FBXZmfrmZyqJ6l/7QylPPro6jz55PqcP389H/32aF56cXvm5hrp6V1Z6IfBLVu356OP/mc2bdySZquVudnZXL8+88dnv1o7kLR3dGVgoOOWkHDjdpXzWbGiJ+fOz6ZS/vYVEXdSyo3vV6vVyJkz1/LSS31JktOnR3P8xPk8s2djenru/BP27u72PPXUhiwsNHPk6OX8/vcX8td//ZN7nuHH4sqV0Xz++YfZs2ddBge33/E17e1t2bVrbR57rJUzZ67m6NGpPPvsazl37kzeeedUSqVWWq0bqx9KpWpGRnbm9dc3LPmSFQAeTkIDACzCzOx0urtWZWGhkpmZyfR03/jQW0oplXLlrj9cl8uVVCqlTE3O5c/+dEfa22/80dzVWUtX5+2XC3xl/br+9Pa25/0PD6ezsyePP1HsB+n29va01/pz/fpM+vo609l54393o15vpKOjmvXrB7L/96MplRd3XX+5XM7x4+OptbelVCrl9OnRnDl7Ma/u3XpXH4ArlXJGRobS1lbLBx+8nb17X3/gPjhfvXol+/d/mFdf3ZJa7bv/WlcqlbJx44r09nbmd797N6+++qd55BF3kgCgWA/Wn7YA8ANpNW9cItFWq6WxUMp8fWFR52k0mllYaOWxXcM3I8Pd6uvtyBOPD+f8hWvp6+tb1Nf/NrsefyqffHou9Xt8b41GM5VKOQt/POzs2cXdVnFycjbHj4+nWq1kbGwyx4+fz4svjNx1LFhoNjM9Xc/IyNps3tyRzz//eFFz3K/m5uby/7d3p09xnXfax6/Te0PTTbPT7JvQbm2WZK1WnGQmiu1UUql5NX+ZX8xUzYuZeuqpSSqVPJ5x4sSSrQW0IUvWDmIHAQ00DU3v5zwvkLAxArEchGR9P1WpirtP3313Y9fpc537/v26uq7qgw8a1xQy/FBxcYHee69CnZ2Xtmh2AIB3GUEDAAAb4HC6lM+bMiQFg8VKzOfWHTZksjnFZ9PyeNxyOJzK/mi7hWVZymRymp/PKJvNL+tskc7k5PV61dZaoYnx8c1+pGWKioq0d+8xXbnap0wmJ9M0lUikNDk5+7wbhLlQ3NKylM+bMs2FJpIul0OxWFJO58JWjsloSv390+t67/hsSteuD+nEyd3K5w1du/ZEBw+ufSl/Pm8qHk+pqKhYDoeh+vpSzc1FlU5vvgXom6Kn54l27SqT17uxLTOlpQEVFuY1NTVl88wAAO86tk4AALABgUBIMzNJlZYG5DAMhUJhxeMxZbN5+f3uFbtPWJLy+bzm59NKpfKSsVC8sKS0SNPTCbndTuVypp72Tmp+PiOfzyWn06FczlQqmVNJiV9NTWULKwZMh0KhsFo9Bbp954EqKitt/5zlFRUqGWvU//m/V+X3OxUIeOV2OxcDEKfD0O5d1aqqCsqyLMVmkrr/IKpsRotbLfbta1R//5imp4fV3l4u/ypbQnI5UwMD0+rrj+mD47vk87lVWBhSOj0nSwuBhtO5cthgyVI6lVUylVNRUfGSYppNTWE9ffpEu3btte8L2iaWZenZs37t2tWyqXFaW8v14MEDHTt20qaZAQBA0AAAwIY0N+/Qg/uXVVoakKTFsCGdTiseT8jhkPw+l5wuh4znLSqzmawS8xmZ5kLBQp/Pq1Qqo3g8o0OHSjQ+MaebXUMqKvKqtbVM4WL/9/UNLMm0TEWj87p7b1S5rHTwYIsMQyoo8MrMzyuZTK65jsJa9PQ80a1bV1VW5tOvfrVL4eICmVZ+odWmw5BhGJqYmNPduyO61TWoggKvqiOl2ru3Xt9+O6DhkYVVFg6HoSNHWjU+HtOtrlHJMtXSUqLiYr88HqdyOUvz8xn19k4pFkurvqFCZ07vlcvl1MBAVA6HV/v3l8jnK9JcIiHLMuX3u+RyOeVwGLKshZahqVRW2awlr9evUCi42AHjhZqasP7xjx7t3LlnQ+023yQjIyOqqirc9OcIBv1Kp0eUTqdpXwkAsA1BAwAAGxAMBpXOOBe2LzzfH29I8nm98nm9yuXzSiXnlU/lZFmWcrmcDMNSKFgol2vhjvzsXEoT0YRqa4vVPzCloaGYzp5ultvtlGlZyuVMGebCheTC9aRD5eVBVVaElExm1NHZrb176lVeHlJdbUCjo6Nqbm625fPdunVDw8MP9MknC6sKpIWLeeUXWlWapiSZCocLdPZsm5LJjP7xjyfy+TyqqytTcXGhrl97LElKZzIyDEOVlWFVVoY1P59Wb++Y+vvHlcnk5HQ55PN51FAf0YEDgSUXz729MVmWUzU1YRmGIY+nWPm8qVQqqXQqJ9MyZRgL7US93oAKC90rttI0DEPhsFfxeFyhUMiW72m7jIz0a9eusC1j1dQEND4+rrq6OlvGAwCAoAEAgA1qbtqlnp5H2r1r+ZYFl9OpQKBIkjSXSMjtNlRYsHTLgJnPq7c/ptaWCj17NqOTJxoXL7IdhiG3y6lc3pTL5ZKhpVfPfr9Hp0426fKVXh14r0k+n1vTsaTscO/eXY2NPdL587vldC5sPTBNU/l8Xq7nKzR+rKjIp1//eo/+938fyOt1a8eOiHLPS1ZkMkmlM2l5PQt3zAsKvNqzp/6V85iamlNBQVjzydkl4YPT6VBhYeGGPpvP5/pJ1GlIp1Py+YptGcvrdSmVStkyFgAAEsUgAQDYsJraWsVmXBp9Fl/xmHQmI9NMLwsZLMtS17fPVFkRVF/fpI6+X7dsGbxhSE6noVwu99Kx3W6nPjjeqBu3ni5uH9iseDyuBw9u6qOPdi6GDJZlrRoyvOByOfSLX+zUt98+VSqV0Y4dEUkLF7Lzidl1zS+ZzKjr9jPt239IC5Ut7OFwGLZ8T9vNsqxlW0M2auE72VjXFAAAXoagAQCADTIMQ0ePndKT7nkNrdDCMZlMqLBw6d53y7J0q2tUqZSlVCqn3bsqV+ym4DCeb1WwXn6x7fW6VF8X0vDIlDwe3+Y+kKQbNzp06FCtPJ7viyiapvm8JsOrX+/1OrV/f0TXr3crEFiYz51vh+VwSsnk9ysucrmc0um0UqmU0um0stmsrOeBwtxcSleu9uv990/L5/NJsq+eQiaTl8fz8mKUlmVpampKIyMjGhgY0OjoqOLxlUOk7eTxeJTJvDyAWq+F72Tz/+4AAPACQQMAAJvgcrl08tTPNDwiXbsxpFhsfvG5XD4vw7DkfH7n2TQtDQ3FdOHrPpWUhNXSUq3x8VmVl6++DeBVd5ybGks1ODip4vDm9uybpqnJyRE1NpYsPvZipcR67p63tJTp2bMpmeZCcNDeXqfr14d07/6g4vFZTcemND8fVz6flGWllM8nlUrPauzZmG7c7FHntWEdP35OwWBQklRQENTMzPxqb7lmsVhKRUVFSx7LZDJ69OiBvvrqfzQw0KVEolu5XL9mZx/r0aMOXbjwhfr6et+olRDFxWWamJi1ZayJiXmFN/nvDgAAP0SNBgAANsnpdOrY8dOamZnRkyf3NTfbp9qaQkkZ+f0Ozc6mFYulNDY+r5qaUp04sUder0u9vWOqqip6ZecAh2Eob5myZC2r1SAtbFkIFHnlcm3utH7v3ndqbAwvWV1hWaYMQ2tazfDD+dTUhDQ4GJUkFQX9OnGiXRcvfqcv/35PkUhINZEieTwuWZaUTuc0PDKjfF4KBArlcGQ0ONiv9vZdMgxDra079eTJDR058uq6DquZmZmXzxeS2+1+/tksPXx4T+Pj/WpqCuvDD5teurIkm82pr29QX311T+3t76m2dvuLJjY1taij40vV1ZVuapxsNqf5eb31xTEBAG8WggYAAGwSCoV05MgHymazGh0ZUdftTrW1Fsvrcau8Iqg9e4NLQoXp6TlFIiFlc3m5Xc5VRtZii8yXXfCn0lk1NZZrMjqh4uKNFwgcGRnQwYNL72xvtBZAfX1Yd+5MSJKSyZQ8HqeOHm3U8HBCDQ0VmozGlUjknneScOu991pUWOhbfM/790fV1TWngwffVzgc1tycqWw2L7d79e9pNd3dUbW1HVl8j1u3rsnvT+rMmZZVwx6326W2tkq1tJTr+vV7SqXm1dravuF52MHj8cjnCyken1cwWLDhcfr6JtXYuMPGmQEAwNYJAABs53a7Vd/QoOLioHbvrlNLa5UqK0PLLmbTmZyCwQIlEtlXljtceOnyo0xroc5DYaBA6czmOgdks2l5PO4ljy2Uhlh/0OD1uhaLWObzGRUV+eT1upXJ5BQKFai5pUo7d9WqfWeNmporF0MGaaH2xZ49ERUUJHXv3h1JUlNTux49erbhzzY3l9LcnLW4ReDu3dsKBNLavbv6lStKXnA4HDp6tFGTk70aHBzY8Fzs0ta2W/fuPZO1Qv2OV8lmc+rvj6uubnMrRQAA+DGCBgAAtsirLl8Nw5DL5ZDH41IikV53bwVLluLxlAoLgzKej7d59nR4+OHFb2GhZ3Fu65lie3ul5uZGNT09rfr6Bs3P+zQwMLnuuaTTWXV2Durw4ROSpKmpKSWTY2pvX96W9FUMw9D77zfoyZNvlc1m1/16O4XDYRUX1+n+/dF1v9Y0TV292qf33ju2YiFSAAA2ijMLAABbxJKx6t1mj8eldDqnggKvDMOhubmVw4YfryzIm6ampxOyLEPpdFoz8bgSifkl72dZlqLRqO7fv6Pbt6+rq+ua7t27o/HxsZfOy+PxKZ1eevH8YiWFaUr5vKlc7vv/5fOWVvp4qVRuccvFi5Ahnc7K41n7rk3DMLRrV6W6ux/IMAwdOXJcw8M5dXePr3mMubmULl3q06FDJxQIBCRJT57c165dVWse48ccDodaW0vV29uz4THssnPnHuXzId2+PbjmlQ2ZTE7ffNOj1tb3VF5evsUzBAC8i6jRAADAFiktqdTY2Iyqql5eN6GyIqSh4RntDvpUWOhTMplRLJaU1+OUz++W4/kFuqXv6zNkczklEmlls3kVFJ8LHckAACAASURBVHrlfb7VYWxsRgUFlv7xj89VWVknl8ul0dF+hcNuRSJBeb1uGYaUTqc1OnpP3313U7W1zWpsbF5s99jQ0KKenvuqrFwoDPjiujWXM+Vw6HmLy+/DDsuylMuZMgxDTqdjyWqF3t5JlZYWaGQksfjY0NCMqqvL1vUdhkIFSiZHlc1m5Xa7dfz4ad27d0cXL3aroaFYdXUlcjqX3zeJxRJ68iSqZNLQsWMfLoYM6XRa6XRcweDmLrBra8O6cOGp2trabVpJsjGGYWj//kPq6Xmir756rLq6IjU2lsrtXv4Tb24upZ6eqKamMtq//7hKS9f3twAAYK0IGgAA2CItre26dfOrFYOGSKREDx8Na9fOChmGIb/fI5/Po3Q6q3g8LcOQnE6HLMtSPm8pl0vJMAwFAj6Fil2L6xvS6ZxkOXX4cKOmpuK6evWuampCOnasTgUFvmXvW14elGmaGhqa0jffdGv//qMqL69QW9sO3bnTqXw+L4fDoVwuJ4djoYuEYRgv2fZgyOF4ETgsvMbpNJRO5zU2Nqfjx5t19+5CQciF1pnz2r9//d0NGhuL1df3/UX93r3vKZvdrf7+Xl282KNAwCm/3yWHw1A2ayoWSykQKNWOHceWdVPo63uq5uaNF8x8weFwqKzMq2g0+kasCmhpaVNTU4uGhgZ05cpjeb2mCgo8cjoXvpPZ2bRcrkK1tu7X/v1l2xqOAAB++ggaAADYIn6/X4ZRoPn5tAoKvMueNwxD1VVhjY7GFYmEnj8m+Xxu+Xxu5fOmTNNSfDYly7RUXFwol2v53fuep5Nqbq7SxMSM7t3r189/3i6Px6l4fF6plOTzLQ8bHA6H6utLVV1drI6O68pmDygSqVFlZb26u6Nqbl5YKbAQJEgL6yqWX5wutL40ZBhO5fML2ymePBlXdXV4yVwHB6dVV7exC9ySkkI9ejS95DG3263W1h1qaWlTIpFQJpNRPp+Xx+NRYWHhiq0+4/Fp1dYG1j2Hl8/Lr3g89kYEDdKLv2mj6usblUwmlUqllMvl5PF45Pf7F1euAACw1ajRAADAFtqxY49ufzuy4v755uYqPXw4oWw2v+w5p9Mh07SUz+VVUvLykCGRyOjZ6JyKgn7dvdunkyeb5fW6ZBiGgkGfUql5ZTIrFy10u506caJZjx93aWpqUvv3H9TNm/3PVyh8f5xlacV6DJIWV18kEmndvTuqvXtrF0OFTCav7u5JNTZWrDzAKtxupzKZ9ArvaygQCKikpETl5eUKhUIrhgySlM1m5PFsvEXmD3k8rhXntd38fr/C4fDid0LIAAB4nQgaAADYQmXl5aoob9Ht20MvDRt8Prf27m1QR0e/8nlzyXPZbF4z8aSKiwsXCyv+UCqVVUfngI4ea9d33/Xr6NEGud3fX0S/CBsSidlVQwKn06EPPmjUd9/d0tBQv3bsqNVf//pQ6XT++TgLY1nWysUfJSmdzujvf3+sggKPXC6nXnSwuHNnWAcPNr+0bsBamKYlp9OecMDpdC77njcqnzfldLI4FACAH+Ps+Bbq6urS9evXtXfvXp04sdCqK5/Pq6OjQ93d3crn84pEIjp16tRi8SsAwPZpbWvX40emrl3r06FDtcsuuCsqQsrnTX1zqVfHjtbJ5/Mok8lqfj4vt8v50jvwsdi8bt4a0ZEjbc9XPuQVCCzfnuFwGHK5DGWzWXmeF458Ga/XLa83r+HhXv3ylztVVOTT//t/9/TRR20KhfzP6zO86KKxvF7D9HRCf//7Ex06VK9s1tTYWFyWtbCSoqWlWiUlRev92hYlkxl5vf4Nv/6HvF6fksmsfL7N3+FPJrPyepdvS8H2mZqaUk/PA80nZvViu4/L5VFDQ6siNbW08gSA14Sg4S0zPj6uhw8fqqSkZMnjV65c0cDAgD766CP5fD51dHToiy++0G9/+1tOqgDwBtjRvkujIyFduXJXhYVSW1uZQqGCxeerq8PyeJz66kKPZJnatTui8rIiuVzfb6kwTUvDwzPq7Z2S1+fTiRO75fd79N13/WptXbmDgN/vUSIxL49n9UKMRUVuuVx5GYah9vYaFRUV6O9/fyifz6m9e6tUWxv+0coGSwMDMX333YiyWVOnT7epoqJIc3MpffHFI+WfTz0c3lzo3dc3pcbGw5sa44Xa2ib1999WOFy46bGGhmZ1/HjEhllhMyzLUl/vU/X3P1JRQGpvLlZRUcUP2qrm1Nv/UF89uq2qqga1tu2U17s8lAMA2Ieg4S2SzWb11Vdf6fTp0+rq6lp8PJPJ6NGjRzp37pxqa2slSefOndN//ud/anh4WHV1dds1ZQDAD1RHIqqORBSLxfT4yX0lEmMyDOt5iUVDlpzaveu4SsvK1dvbrZs376i8vGDx+bxpKVJdqmPHd8nr/X51wsTEjPbsaV3xfV0uhywrJ9O0XroF44Wpqbja2koXW2lGImH97ncfaHp6TjduPNG1awNyuZxyOBbChkwmr8JCr957r15FRQt39mOxlNxuryKRCs3NSTMzQ5v6znK5vOLx/LKAfaNKS0t1925WuVz++faOjYnH5+XzUftgu+VyOXVcvajy0rxOHa946d/U63Vp545ytbdZGn02rUvf/FXvHz2rYDC4DTMGgHcDQcNb5NKlS6qrq1Ntbe2SoGFiYkKmaS6GDJJUWFiocDissbExggYAeMMUFxfr/fcXtr69qNvw424M+/cfVCz2TGfONMuyrFW7NTgcxiu7OTgcjsVtDysxTVM+n3vZ+4XDAf3iFweXHPfj1XIvwokXQiG//P5i9fVtLmjo759UQ0Pbpsb4IcMwVF/fpoGBZ2pu3ni3iCdPompttWeVBTYmn8/r8qW/a9cOvyrKXx1EGYahSHVI4WK/rl77SkeOniNsAIAtQtDwluju7lY0GtVvf/vbZc8lk0k5HI5lywD9fr/m5+fX9T6xWEzRaHRTcwUA2COZTCsanV31GNO0lE7nFI0mVj0ukUhrft7x0s4V379fVtPTSaVSLjmd629DuXSsjCwrJ0kaGJjc0BiJRFqPHk3p8OEdtp6bCgsDunUrKpfLoYKC9a9ImJyc0/R0Xg0NJufMbXT3zi1VlVtyGI5X/vv/Yztag7r0zZc6eOgDud0r1y4B8OaJxWLbPQWsAUHDW2Bubk5Xr17V+fPnV23Z9TLr7Vf+9ddfy++3p+AWAGDz+vvH13RcX989m94vbss4z0eTJN240a8bN/o3PMrQ0J/smtCPxl3bd7uSp08HbJoJNupJ9+ZePzT8Z3smAuC1SSaT2z0FrAFBw1sgGo0qmUzqD3/4w+JjlmVpdHRU9+7d0/nz52WaptLp9JJVDclkUpWVlet6rzNnzrDVAgA2IZFIaGioT/H4tL7vyGjINA1VVERUXV2z5juoN25c0aFD1avWVZCk69ef6MiRulXD5dnZlAoLg6uO1dX1VPX1xSopKVnWVWK97twZUUVFs65cuaLTp09rcPCpysvdikTCq66WsCxL4+Nx9ffPat++w1safs/Pz+u7726qoSGkioqiVb+/XM7UyMi0otGc9u07xF3wbfbgwR3VVJoKFm2864dpWbrRNaH3j55e940ZANtncHBwu6eANSBoeAtEIhH9/ve/X/LYxYsXFQqFdODAAQUCATkcDg0NDamlpUXSwo+n6elpHTt2bF3vVVxcrLKylSuXAwBebnIyqvv3v5XHk9POnaUqLa1a8rxpmhocnNLjx7dUWFiiffsOvbLyfXPzDqVSU6qvL131uLq6UuXzeVVVvbyrRC5nyu12KxRafT96TU2p5uZSam/feCtKScpkcnK5/KqqWvgOysvL1d7ert7ep7p//4nCYbdaWsoUCPgWL/BSqYx6eqJ69mxekUijfvGLM6/lYr6qqkrd3Y91506/qqsL1Nxcttj60rIszc6m1NMTVSyWVX19mw4caKab0zbL5XLK5+bU3LT5GyN1tQmZ+bwqq6pefTCAN8Ls7OpbCvFmIGh4C3g8nmXVtl0ul3w+3+Lj7e3t6ujokM/nk9frVUdHh0pKSlRTU7MdUwaAd8rgYL/6+r7T0aMNS7pB/JDD4VBDQ5kaGsoUjc7q0qUvdezYWQUCK7d+bGpq0ZUrPa8MGlpaqnXz5uMVg4ZkMiO//9UtJmdns4rFsq8sPvkqvb2TampqX/KYw+FQS0urmptbNDk5qQcPHimVGl8sUOnxeNXY2K7du6tf691lj8ej3bv3ateuPRodHdGtW93K5dKL34HPV6iWloM6eHD1vwFen2fPnilStfGVDD/UWB/Sw+4eggYAsBlBw0/EBx98IIfDoS+//FK5XE41NTX68MMPuesCAFtsaGhQQ0P3dfLk2u90l5UV6fhxjzo6LurEiZ+tuD3A7XaroCCs6emEwuHCFcfz+z1yOJyam0srEFi6SsI0LeVylgKB1VcHpNNZpdNO1dQ0aXQ0pkgkvKbP8mOmaWp4OK4PP6zV1NTUsucNw1BZWdkbt3rOMAxFIjWKRAjo33TpdEp+nz0/Yf0+j9KpaVvGAgB8j6DhLfXJJ58s+WeXy6WTJ0/q5MmT2zQjAHj3zM3Nqbv7W50507LuYLew0KsjRyK6du0bnTnzixXv4u/Zc0BXr/5Dp041rrhaQpL27WvUtWuPdfp0i9zuheIQlmUpHk+psLBo1ZoL+bypq1f7tH//BwoEinTp0pcKBv0KBNZ319iyLN24MaC2tv0E3dgyZj4np9ueVS9OpyHTNG0ZCwDwPX4FAACwQU+ePNC+fVUbvqgOhQoUCjk0Obly+8eCggIdPnxSly/3KZnMrHhcUZFf+/Y16vLlp0qnc4shg89XII9n5YAim83p8uWn2rHjoEpKSuXxeHTs2Fl1dg4pHl97ZW/LsnTr1oBCoXrV1dWv+XVvokQioampKU1MTGhmZkbZbHa7p4QfcHt8ymTztoyVyebldq+/xSkAYHWsaAAAYANyuZxisTEdPNi2qXFaW8v13Xf3VVZ2ZsVjiovDOnLkjK5c+UYNDUE1NpbK5XIuO668PKR9+5r05ZePVFMT0o4ddfL5Xr4qwTRNDQ1N68mTKe3ff1Tl5RWLzxUWFurEiZ+po+Oiqqv9amoqXXE1hWVZGhub0cOHE2ps3K3GxuZ1fgNvhlwup4H+PvUPPFGB31BBgUtOh0OZrKl4PC2vL6S2tt3Laibh9QsGg+p7mlWjDXnW1FRCweDGtgkBAFZG0AAAwAYMDPSrvv7lxRfXIxDwKZsdViqVWjEUkBYurj788J81ONivS5eeqKjIodraoHw+twzDUDqd0/j4nJ49m1dj4z45nW51dvapuNitmprgYlCQyeQ0OjqraDSl2tpmnTnz/ku7O/j9fp09+0uNjAyrs/OhvF5T9fWh5/UgHMpkcopGExoenlVlZZ2OHftoS1tRbhXLsvTo4X2NPutVQ31Qp0/WvjTEic8m1d19Q3fuWDp48LhCoc3/7bExJSUl+va2pVzOlMu1ucW5Pb2zOvz++jp0AQBejaABAIANGBsb0qFD9nQiqK4OKBqNqra2dtXjnE6nGhub1djYrOnpaY2ODiuTScmyTHk8PpWU1Czp2tDevlOTk5MaHx9RJrPQScHj8amqql779lW8sruDw+FQbW2damvrFI/HNTw8qPHxlPL5tDwen4qLm3XuXM1bW49hoaZEhwKFKX14pmnV7yNY5Nehg7VKJjPq6Pxae/ceU3lFxYrHY2s1NOzQwGCfmps2/t9gYj4thzPwVgZkAPCmI2gAAGADstmMPB57TqNer1OZTHpdrwmHwwqHV1/ybWeHh2AwqGBwz6bHeZN8e/uGwsUZtbZUrvk1fr9Hp0426tLlDh06fJaVDdukvqFRFy/cU0N9WE7nxoKuh48m1dp6xOaZAQAkikECAPBGsKztnsG7ZXRkRKY5pdaW8nW/1u126oPj9bp547Is/nDbwuVyadeuw7p2c2RDf4Oe3ik5XOWqqFx7yAQAWDuCBgAANsDt9iiTyW349aZpKZlMaX5+XvF4QhMTE4pGo1y4viY9PQ+0e1fVhl/v83lUXu7R+NiYjbPCekRqalUV2aWr14aUz6+9ReWjJ1FFp706cPD9LZwdALzbCBoAANiAsrIqPXsWX9drLEvKZLKamZlRPD4tKS2nM6eJiZjKytIaGrqtCxf+V93dj5XLbTzEwOoSiYSczox8vpXbfq5Fa0uZunse2DQrbERTU6uamo/owqUhPXoSXTH8M01LQ8MxfX15UFmzXEePnnxljRIAwMZRowEAgA1obGzWlSvdamhYWzE607QUj8/I6bRUWOiWy+WRJCWTGbndHrW2Ltxdz+dNDQyM6+LFx9q9+5CqqyNb9hneVd3dD9Xasvk2lX6/R4aSSiaTFBTcRtWRiCqrPtHw0KA6bjyUz5tTKOiR220on7M0n8wrOpVVTU2zjn1wQl6vd7unDAA/eQQNAABsgNvtVkFBiWKxhIqLC1c91jQtzcxMq7DQvayAZE9PVC0t3y/hdzodamoqV319qTo7byudTqmxsXlLPsO7aiY2qf17a2wZq7TEr5mZGYKGbeZwOFRX36C6+gbNzs5qbm5O2WxWvkKXSit92n8wzAoGAHiN2DoBAMAG7dixW3fvPlu1roJlSTMzMQUCnmUhQyKR1vh4QpWVxcte53Q69MEHzRoaeqhnz0Ztn/u7zbLtotPtcSiTydgyFuxRVFSk6upq1dfXKxKJqKSkhJABAF4zggYAADYoFAqppmaHbt0aXDFsSCbn5fM55HY7lzyeTufU2dmv999vW/EiyDAMHT/eqHv3blEk8g1lmZYcDn5OAQDwQ5wZAQDYhObmVgWDdbp2rW9Z5XvLktLp5LKig7OzKV269FQHDzarqGj1Jfcul1PV1QUaHR2xfe7vKsPhVC6Xt2WsVDrPnn8AAH6EoAEAgE1qa9upurq9+vrrPt2+Pai5uZQkKZPJyO12yjAMWZal8fG4rlx5qjt3nun48Z0Kh4vWNH5zc5l6eh5u5Ud4p0QiDRoYnN70OAt/06RKSjZfWBIAgJ8SikECAGCDSKRW1dU1ikajunv3gdLpIaXTcwqFPMrnTaVSeZWXh3TgQKsKCtZ3B9zn88jlytLdwCYNDU365ptHam4q29Q40ck5lZRUy+l0vvpgAADeIQQNAADYxDAMlZeXq7y8XJZl6cKF/9F779XJ7XbK6XRsqiBdMOhVIpEgaLCBy+VScahCk1NzKi0JbHic7u4p7X/vjI0zAwDgp4GtEwAAbAHDMORwOOT3e+RyOTdd9d7lciibzdo0O7Tt2K07d8c2XKthdHRGDmeRCgtXb20KAMC7iKABAIAtYmdLvXzeksvFQkS7BAIB7d71vq529K87bJiIzurRk1kdOfLBFs0OAIC3G0EDAABbxDBcymZztoyVSGTk8/lsGQsLKquq1NZ2SJcu9yk+m3zl8aZpqufphB48nNPJk+eozQAAwAq4NQIAwBapq2tWX9+g2toqNzVOLpfX3JypoqK1danA2lVVR1RQGNCDB3eUTj9Tc2NIkUhYDsf3q1GSyYy6eyY1MZFSTW2zTp1ql8PBvRoAAFZC0AAAwBapq6vXxYv31NpasaltFP39k2poaLNxZvihYDCoY8dOKZ1Oq7e3R99cGpRhmLIkGTLkcvvU0rxHe/dV2rodBgCAnyqCBgAAtojT6VRpabUmJmZVURHc0BiWZWlgYEanT1MPYKt5vV7t3LlbO3fu3u6pAADwVmPdHwAAW6i9fY/u3h1TKpXZ0Ovv3BlWXV0bhSABAMBbg6ABAIAt5PP5dOTIaV250q9kcu1hg2VZundvRFKxWlvbt26CAAAANiNoAABgi4VCIR05ckZXrgyovz8q0zRXPX5mJqGOjl45HOXav//Qa5olAACAPViHCQDAaxAMBnXmzC/V1/dUFy70qLTUq8bGsHw+jxwOQ5lMXpOTc+rtnVZBQVi7dn2g4uLi7Z42AADAuhE0AADwmrjdbrW1tau1dYcmJsb19Gm/0uk5maYpt9uj4uJSnTjxvtxu93ZPFQAAYMMIGgAAeM0Mw1BFRaUqKiq3eyoAAAC2o0YDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwDUEDAAAAAACwjWu7J4BX6+rqUl9fn2KxmJxOpyorK3Xs2DEVFxcvHpPP59XR0aHu7m7l83lFIhGdOnVKgUBgG2cOAAAAAHjXsKLhLTA6Oqrdu3frN7/5jX7961/Lsix9/vnnymazi8dcuXJFfX19+uijj/Tpp58ql8vpiy++kGma2zhzAAAAAMC7hqDhLXD+/Hm1t7erpKREpaWlOnv2rObm5hSNRiVJmUxGjx490vHjx1VbW6uysjKdO3dOU1NTGh4e3ubZAwAAAADeJWydeAtlMhlJktfrlSRNTEzINE3V1tYuHlNYWKhwOKyxsTHV1dW9ckzLsiRJQ0NDWzBjAMC7KBaLKZlManBwULOzs9s9HQDAT8CL65UX1y94MxE0vGUsy9LVq1dVVVWlkpISSVIymZTD4VgMHl7w+/2an59f07gvfgCeOXPG3gkDAAAAgM1mZ2cVCoW2expYAUHDW+by5cuamprSp59+uqbjDcNY03GRSESDg4MqKipa82sAAAAA4HWyLEuzs7OKRCLbPRWsgqDhLXL58mX19/frk08+WdJNwu/3yzRNpdPpJasaksmkKisr1zS2w+FYsvUCAAAAAN5ErGR481EM8i1gWZYuXbqk3t5effzxxwoGg0ueLy8vl8PhWFJfYX5+XtPT02sOGgAAAAAAsAMrGt4Cly9fVnd3t375y1/K7XYv1l3weDxyuVzyeDxqb29XR0eHfD6fvF6vOjo6VFJSopqamm2ePQAAAADgXWJYlOt843322Wcvffzs2bNqb2+XJOVyOXV2dqq7u1u5XE41NTU6derUki0WAAAAAABsNYIGAAAAAABgG2o0AAAAAAAA2xA0AAAAAAAA2xA0AAAAAAAA2xA0AAAAAAAA29De8h3T1dWlvr4+xWIxOZ1OVVZW6tixYyouLl48Jp/Pq6OjQ93d3crn84pEInSwAACsSVdXl65fv669e/fqxIkTkjivAADWL5FIqLOzU4ODg8rlciouLtaZM2dUXl4uSbIsSzdv3tTDhw+VTqdVUVGhkydPqqSkZJtnDokVDe+c0dFR7d69W7/5zW/061//WpZl6fPPP1c2m1085sqVK+rr69NHH32kTz/9VLlcTl988YVM09zGmQMA3nTj4+N6+PDhsh95nFcAAOuRTqf1pz/9SQ6HQ7/61a/0L//yLzp+/Li8Xu/iMd9++63u3r2rkydP6re//a38fr8+//xzZTKZbZw5XiBoeMecP39e7e3tKikpUWlpqc6ePau5uTlFo1FJUiaT0aNHj3T8+HHV1taqrKxM586d09TUlIaHh7d59gCAN1U2m9VXX32l06dPL/khyHkFALBet2/fViAQ0IcffqiKigoVFRWppqZGwWBQ0sJqhrt37+rgwYNqampSSUmJzp07p1wup+7u7m2ePSSChnfei8TvxY/CiYkJmaap2traxWMKCwsVDoc1Nja2LXMEALz5Ll26pLq6uiXnD4nzCgBg/fr7+1VWVqa//e1v+o//+A/993//tx48eLD4/OzsrJLJ5JJzi9PpVHV1NeeWNwRBwzvMsixdvXpVVVVVi8tck8mkHA7HkrtRkuT3+zU/P78d0wQAvOG6u7sVjUZ19OjRZc9xXgEArNfs7KwePHigUCik8+fPa9euXbpy5YoeP34sSYvnD7/fv+R1fr9fyWTytc8Xy1EM8h12+fJlTU1N6dNPP13T8YZhbPGMAABvm7m5OV29elXnz5+Xy7W+nxWcVwAAL2NZlsrLyxcD7LKyMk1PT+v+/fvasWPH4nGcR95cBA3vqMuXL6u/v1+ffPLJkqrffr9fpmkqnU4vufuUTCZVWVm5HVMFALzBotGoksmk/vCHPyw+ZlmWRkdHde/ePZ0/f57zCgBgXQoKCpZ0xZOkcDis3t7exeelhZUNL/6/tHBu+fEqB2wPgoZ3jGVZunz5svr6+vTJJ58sFlR5oby8XA6HQ0NDQ2ppE2dh5gAAB7NJREFUaZG08B/w9PS0jh07th1TBgC8wSKRiH7/+98veezixYsKhUI6cOCAAoEA5xUAwLpUVlZqZmZmyWOxWExFRUWSpKKiIvn9fg0NDamsrEzSQivl0dHRl27jw+tH0PCOuXz5srq7u/XLX/5Sbrd7cX+Tx+ORy+WSx+NRe3u7Ojo65PP55PV61dHRoZKSEtXU1Gzz7AEAbxqPx7OsnaXL5ZLP51t8nPMKAGA99u3bpz/96U/q6upSc3OzJiYm9PDhQ50+fVrSwpaJffv26fbt2wqFQgqFQurq6pLL5VJra+s2zx6SZFiWZW33JPD6fPbZZy99/OzZs2pvb5ck5XI5dXZ2qru7W7lcTjU1NTp16tSSLRYAAKzkz3/+s0pLS3XixAlJnFcAAOvX39+va9euKR6Pq6ioSPv27dOuXbsWn7csSzdv3tSDBw+UyWRUUVGhkydPLgu/sT0IGgAAAAAAgG1obwkAAAAAAGxD0AAAAAAAAGxD0AAAAAAAAGxD0AAAAAAAAGxD0AAAAAAAAGxD0AAAAAAAAGxD0AAAAAAAAGxD0AAAAGzx2Wefqa+vT5I0Ozurzz77TNFo9LXP48KFC/riiy9e+/sCAIAFru2eAAAA2DoXLlzQ48ePJUmGYSgQCKipqUmHDx+W2+3esvctLCzUv/7rv8rn8615nul0Wv/0T/+0ZXMCAACvB0EDAAA/cXV1dTp79qxM09SzZ8908eJFZbNZnT59eslxlmXJsiw5HJtf8OhwOFRQULDpcQAAwNuHoAEAgJ+4H170t7a2amRkRH19fWppadFf/vIX/epXv9L169c1NTWl8+fPKxKJqL+/Xzdv3tT09LQKCgq0Y8cOHTx4cDGEmJmZ0cWLFzUxMaGioiKdOHFiyXvOzs7qv/7rv/S73/1OZWVlkqSpqSl1dnbq2bNnkqTS0lJ9+OGHevz48eKqi88++0yS9PHHHysSiSiRSOjq1asaGhqSYRiqqqrSiRMnVFRUJEkyTVOdnZ169OiRDMNQe3u7LMva+i8VAACsiKABAIB3jNPplGmai//c2dmp48ePKxgMyuPxaHBwUP/4xz908uRJVVVVKR6P65tvvpEkHT58WJZl6a9//at8Pp9+85vfKJvN6sqVK6u+ZyKR0J///GdVV1fr448/ltvt1tjYmEzT1HvvvadYLKZsNquzZ89Kkrxer3K5nP7yl7+oqqpKn376qQzDUFdXlz7//HP9/ve/l9Pp1J07d/To0SOdOXNG4XBYd+7cUV9fnyKRyNZ9gQAAYFUUgwQA4B0yPj6u7u5u1dTULD525MgR1dbWKhgMyufzqaurSwcOHNCOHTsUDAZVW1urI0eO6MGDB5Kk4eFhxWIxnTt3TmVlZaqurtbRo0dXfd979+7J4/Ho5z//ucrLy1VcXKz29nYVFxfL7XbL5XItrrwoKCiQ0+lUd3e3JOnMmTMqKSlROBzW2bNnNTc3p9HRUUnSd999pwMHDqi5uVnhcFinT5+Wx+PZom8PAACsBSsaAAD4iRsYGNC//du/ybIsmaaphoYGnTx5UtPT05Kk8vLyJcdHo1FNTEyoq6tr8THLspTP55XL5TQ9Pa1AIKBAILD4fGVl5apzmJycVFVV1brqP0SjUcXjcf37v//7ksfz+bzi8bgymYzm5+eXvLfD4VB5eTnbJwAA2EYEDQAA/MRFIhGdOnVKDodDhYWFixf7L4IGl2vpzwHLsnT48GE1NTUtG8vpdG5oDht5nWVZKisr089+9rNlz/n9/g3NAwAAbD2CBgAAfuJcLpdCodCajy8rK9PMzMyKrwmHw5qbm1MikVBhYaEkaWxsbNUxS0tL9fjxY5mm+dJVDQ6HY9kqhLKyMj19+lR+v3/F7RAFBQUaGxtTdXW1pIXikBMTE4sFKAEAwOtHjQYAALDEoUOH9PjxY924cUNTU1Oanp5WT0+Prl+/LkmqqalRcXGxLly4oMnJSY2Oji4+t5I9e/Yok8noyy+/1MTEhGZmZvT48WPFYjFJUlFRkaamphSLxZRKpWSaptra2uTz+fTFF19odHRU8XhcIyMjunLliubm5iRJe/fu1bfffqve3l7FYjFdunRJmUxma78gAACwKlY0AACAJerq6vTP//zPunXrlr799ls5HA4VFxdr586dkiTDMPSLX/xCX3/9tf74xz8utrf8n//5nxXH9Pl8+vjjj9XZ2ak///nPMgxDpaWlqqqqkiTt3LlTIyMj+uMf/6hsNrvY3vKTTz5RZ2en/va3vymbzaqgoEA1NTWLKxz279+v+fl5XbhwYbG9ZWNjI2EDAADbyLColgQAAAAAAGzC1gkAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGAbggYAAAAAAGCb/w94bJeNsnk+sQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import matplotlib\n",
    "# force Agg\n",
    "matplotlib.use('Agg')\n",
    "from IPython.display import Image\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "from matplotlib.colors import ListedColormap, Normalize\n",
    "from matplotlib.cm import get_cmap\n",
    "\n",
    "cmap = get_cmap('YlOrRd')\n",
    "\n",
    "def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999',\n",
    "                gridWidth=1.0):\n",
    "    \"\"\"Template for generating the plot layout.\"\"\"\n",
    "    plt.close()\n",
    "    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')\n",
    "    ax.axes.tick_params(labelcolor='#999999', labelsize='10')\n",
    "    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:\n",
    "        axis.set_ticks_position('none')\n",
    "        axis.set_ticks(ticks)\n",
    "        axis.label.set_color('#999999')\n",
    "        if hideLabels: axis.set_ticklabels([])\n",
    "    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')\n",
    "    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])\n",
    "    return fig, ax\n",
    "\n",
    "predictions = np.asarray(parsedValDataOrderTwo\n",
    "                         .map(lambda lp: secondModel.predict(lp.features))\n",
    "                         .collect())\n",
    "actual = np.asarray(parsedValDataOrderTwo\n",
    "                    .map(lambda lp: lp.label)\n",
    "                    .collect())\n",
    "error = np.asarray(parsedValDataOrderTwo\n",
    "                   .map(lambda lp: (lp.label, secondModel.predict(lp.features)))\n",
    "                   .map(lambda x: squaredError(x[0],x[1]))\n",
    "                   .collect())\n",
    "\n",
    "norm = Normalize()\n",
    "clrs = cmap(np.asarray(norm(error)))[:,0:3]\n",
    "\n",
    "fig, ax = preparePlot(np.arange(0, 120, 20), np.arange(0, 120, 20))\n",
    "ax.set_xlim(20, 60), ax.set_ylim(20, 100)\n",
    "plt.scatter(predictions, actual, s=14**2, c=clrs, edgecolors='#888888', alpha=0.75, linewidths=.5)\n",
    "ax.set_xlabel('Predicted'), ax.set_ylabel(r'Actual')\n",
    "pass\n",
    "fig.savefig('temp.png')\n",
    "Image(filename='temp.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}