4. Classify MNIST images with TensorFlow.ipynb


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Introduction: MNIST for ML Beginners\n",
    "\n",
    "In this codelab, we will see how to recognize hand-written digits with a simple neural network running on TensorFlow.\n",
    "This is based on [TensorFlow Tutorial](https://www.tensorflow.org/versions/master/tutorials/mnist/beginners/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%206.04.53%20PM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The full code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "step 0, training accuracy 0.16\n",
      "step 100, training accuracy 0.89\n",
      "step 200, training accuracy 0.91\n",
      "step 300, training accuracy 0.97\n",
      "step 400, training accuracy 0.9\n",
      "step 500, training accuracy 0.94\n",
      "step 600, training accuracy 0.93\n",
      "step 700, training accuracy 0.99\n",
      "step 800, training accuracy 0.97\n",
      "step 900, training accuracy 0.95\n",
      "test accuracy 0.9706\n"
     ]
    }
   ],
   "source": [
    "# import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# define a neural network\n",
    "import tensorflow as tf\n",
    "\n",
    "def weight_variable(shape):\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def conv2d(x, W):\n",
    "    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "def max_pool_2x2(x):\n",
    "    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],\n",
    "                          strides=[1, 2, 2, 1], padding='SAME')\n",
    "\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y = tf.placeholder(tf.float32, shape=[None, 10])\n",
    "\n",
    "W_conv1 = weight_variable([5, 5, 1, 32])\n",
    "b_conv1 = bias_variable([32])\n",
    "\n",
    "x_image = tf.reshape(x, [-1,28,28,1])\n",
    "\n",
    "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n",
    "h_pool1 = max_pool_2x2(h_conv1)\n",
    "\n",
    "W_conv2 = weight_variable([5, 5, 32, 64])\n",
    "b_conv2 = bias_variable([64])\n",
    "\n",
    "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n",
    "h_pool2 = max_pool_2x2(h_conv2)\n",
    "\n",
    "W_fc1 = weight_variable([7 * 7 * 64, 1024])\n",
    "b_fc1 = bias_variable([1024])\n",
    "\n",
    "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n",
    "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)\n",
    "\n",
    "keep_prob = tf.placeholder(tf.float32)\n",
    "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)\n",
    "\n",
    "W_fc2 = weight_variable([1024, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "\n",
    "y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2\n",
    "\n",
    "# define the train step\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=y_conv))\n",
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n",
    "correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    # train the model mini batch with 100 elements, for 1K times\n",
    "    for i in range(1000):\n",
    "        batch = mnist.train.next_batch(100)\n",
    "        if i%100 == 0:\n",
    "            train_accuracy = accuracy.eval(feed_dict={x:batch[0], y: batch[1], keep_prob: 1.0})\n",
    "            print(\"step %d, training accuracy %g\"%(i, train_accuracy))\n",
    "        train_step.run(feed_dict={x: batch[0], y: batch[1], keep_prob: 0.5})\n",
    "\n",
    "    print(\"test accuracy %g\"%accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels, keep_prob: 1.0}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading MNIST training data\n",
    "\n",
    "Preparing the sample image data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Images\n",
    "\n",
    "![mnist.train.xs](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.17.39%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single image = 784 values, ranges from 0.0 (white) to 1.0 (black)\n",
    "\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%206.06.25%20PM.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(55000, 784)"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check MNIST training images matrix shape\n",
    "mnist.train.images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.0509804 ,\n",
       "         0.60000002,  0.86274517,  0.40784317,  0.1254902 ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.0509804 ,  0.85490203,\n",
       "         0.99215692,  0.99215692,  0.99607849,  0.87843144,  0.50980395,\n",
       "         0.0509804 ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.73333335,  0.99215692,\n",
       "         0.99215692,  0.92549026,  0.96078438,  0.99215692,  0.99215692,\n",
       "         0.72941178,  0.32549021,  0.07450981,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.99607849,  0.99215692,\n",
       "         0.95686281,  0.17254902,  0.63137257,  0.99215692,  0.99215692,\n",
       "         0.99215692,  0.99607849,  0.71372551,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.76470596,  0.99607849,\n",
       "         0.92941183,  0.07450981,  0.        ,  0.67450982,  0.99607849,\n",
       "         0.99607849,  0.99607849,  0.81176478,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.38823533,  0.97647065,\n",
       "         0.99215692,  0.66274512,  0.41568631,  0.78823537,  0.99215692,\n",
       "         0.99215692,  0.99607849,  0.80784321,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.8705883 ,\n",
       "         0.99215692,  0.99215692,  0.99607849,  0.99215692,  0.99215692,\n",
       "         0.99215692,  0.99607849,  0.54509807,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.21568629,\n",
       "         0.92549026,  0.99215692,  0.99607849,  0.99215692,  0.8588236 ,\n",
       "         0.99215692,  0.99607849,  0.54509807,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.16862746,  0.36470589,  0.36470589,  0.16470589,  0.09411766,\n",
       "         0.99607849,  0.99607849,  0.81176478,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.09411766,\n",
       "         0.99215692,  0.99607849,  0.80784321,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.0509804 ,\n",
       "         0.86666673,  0.99607849,  0.80784321,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.46274513,  0.99607849,  0.80784321,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.36470589,  0.99607849,  0.81176478,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.36470589,  0.99607849,  0.90980399,  0.09803922,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.36470589,  0.99607849,  0.99215692,  0.18039216,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.36470589,  0.99607849,  0.99215692,  0.18039216,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.36470589,  1.        ,  0.99607849,  0.27843139,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.16470589,  0.92941183,  0.99215692,  0.88627458,  0.18431373,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.47058827,  0.99215692,  0.99215692,  0.95686281,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.17647059,  0.96078438,  0.95686281,  0.27058825,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ]], dtype=float32)"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check MNIST training images matrix data\n",
    "sample_img = mnist.train.images[5].reshape(28, 28)\n",
    "sample_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD/CAYAAADxA2MgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnV2IbNl13//10VXV9dEf92Pu7dEdj4wHtnBATIIjZI9E\nxsQ4xgjGCCRkGSPhMOhhDAL5IdK8DAl5sPIwWFbQgycjMWMsJFvgjByI7QgTBRnsjI0FUqTs+CHj\nmcG5fWfuvd31/V15qF77rrNrn6rqrlMffev/g805dbqrald17/9Ze+211k6NRiMQQraX9Lo7QAhZ\nLxQBQrYcigAhWw5FgJAthyJAyJZDESBky8ku8mRjzC8B+B2MxeRla+0XE+kVIWRlpC4aJ2CMSQP4\nPwD+JYB/BPA6gE9Ya//3lKcxKIGQ9ZEKXVxkOvABAH9vrf0Ha20PwDcAPLPA6xFC1sAiIvAeAG+p\nx2+fXSOEXCIWEYGQaUFzn5BLxiIi8DaAn1CPb2HsGyCEXCIWWR14HcATxpjHAfw/AJ8A8KuJ9IoQ\nsjIubAlYawcAfhPAnwP4XwC+Ya39cVIdI4SshgsvEV4Q+gwIWR+JLxESQh4CKAKEbDkUAUK2HIoA\nIVsORYCQLYciQMiWQxEgZMuhCBCy5VAECNlyKAKEbDkUAUK2HIoAIVsORYCQLYciQMiWQxEgZMuh\nCBCy5VAECNlyKAKEbDkUAUK2HIoAIVsORYCQLYciQMiWQxEgZMuhCBCy5VAECNlyKAKEbDkUAUK2\nHIoAIVsORYCQLYciQMiWQxEgZMvJLvJkY8wbAE4BDAH0rLUfSKBPhJAVspAIYDz4n7bW3k+iM4SQ\n1bPodCCVwGsQQtbIogN4BODPjDGvG2OeTaJDhJDVsqgI/Jy19mcA/DKA54wxH0qgT4SQFZIajUaJ\nvJAx5gUANWvti1N+LZk3e0iI++7l+mg0wmAwiG03b97E22+/jeFwiNFohOFwGGmpVAqZTCbSstms\nO0+n00in00ilUkilUu5cXxPkPHSNXBqCf7ALOwaNMUUAaWtt3RhTAvCLAP7tRV+PREVBBnW/30ev\n14s0uXbz5k3cu3cPg8EA/X5/QiRSqRRyuRx2dnZc04+z2SzS6XREEPS5iIEPB//DxSKrAzcA/LEx\nZnT2On9grf3zZLq1fei7vyAi0O120el03FEaANy9e9eJQr/fj5yn02nk83kUCgXk8/nIeaFQwM7O\nzoSFMBwOkc0++LfwLQJ9nTwcXFgErLX/F8CTCfZla/EFQI6+CLRarUgDgHv37qHb7aLb7aLX60XO\n0+k0isUidnd3J46DwQD5fB7ZbBbZbNYNfnlvbf7rKQJ5+Fg0ToAkREgItAi02200m000Gg00Gg00\nm00AY0tAWwe6ZTIZlMtl10qlEnq9HgaDgXt9mRqIHwGAG/QA3HE4HLpz+R3ycEARWDO+H0COvk9A\nRKBer6Ner6NWqwEYi0C73Y60VquFdruNbDaLvb0997jb7ToBkEE8GAycYxF4IACZTMb9jgx+/Tzy\n8EAR2AB8IfBFwLcEqtUqTk9PAYynA81mE61Wyx3lPJvNotlsBgUgk8kAQKwAiINQrIN0Ou36RSF4\nuKAILJFZy68y0OVu7J+3Wi3UajVUq1XXTk9PXQOAk5OTCRGQo3bw+aRSKfT7fXQ6naDTsNvtOseh\nFgb9WCwE8Rv4foRUKjX1O6CYbAYUgTUgA2MwGLhlP+3Yk2vNZjMiANVqNSIKAFCr1SJTgHa77VYS\nhsOhsyBkORAY3/17vR5KpZITgFCTZUQ5SpPHviD4y4yZTIaWwyWAIrBCQnEAYurLINaDudFooFar\nOR+AfwTGIuA7BEVIRARkfi/BR+JjqNfrbsDncrngUWILJL5AznO5XEQYdAOid3l/xYFsFhSBFRGK\nA5AB2Wq10Gg0UK/X3VHOdfOvAWMRkGVBf5kwk8mg1WpFnIC9Xg+dTgfNZhOFQsENaBn4uvmWgY43\n0NaCDkCSz6ediQCcCFEINg+KwAqYFgcglkC9Xsfp6Wlk3l+v1yPzfL8BYxHQUYT6KFGDQFQAWq3W\nxCAO3e3z+Tx2d3exu7uLQqEwcS6i0O/3kc/n3SDXDkW55h/J5kARWBEhIdCWgIjA/fv3ce/ePdy7\nd29ivq+PEixUq9WCIcMSMajfp9PpTNy5p53n83mUSiUUi8VIk3gDERtZYRAB0EFH8lkpAJsLRWDJ\nzIoD6Ha7TgSq1Sru3buHd955B++8844TAT3f14+BsQj4iUM6gUgLgJ9AJE4+3fS13d1dlEqlSMBR\np9OJWBp6iVEEQK7LZ/UFgGKwWVAEVkBcHMBgMHDTgUajgdPTUycCt2/fRrVajcz3/QbArRLIa+pz\nvVQXaloI/FUAEYFKpeJiDcTp6AuAtgByudyECACgAGwwFIElEkrv1U2Cf8TbH4oF0PN8vwFwYuAj\nd9/QOr1/545rssyoYxh0HIMf16DfO5VKoVKpoN1uO5HQqco6xkA/L3QeB8UkGSgCS2Q0Gk1k9ulj\ntVrF/fv3cXJygmq16nIC5K4rz9Vzfm1+C7Ny/UPz89DvyKCW6zKNkCmEXNeRjNL8Zcp2u40bN27g\n3r17wWmIxBLMslZCn4ckC0VgiYjJH5fgI47A09NTt/6vw3x1erC+4+o7fChKzx8wepqgo/h0VJ+I\ngPgRRqMRut1uRAD8UOZQerMIQLvdBjAOa/b9DtL8IKOQpSD9ZKzB8qAILBE9YHRMvyz7iQ9ALIGQ\nCGiPvy8Agi8AOu03zlfgz821n2I0GrnX6HQ6QQHQPoJpInD//v1IzIGcS+qyH2GoB78+l89Jn0Ly\nUASWiEwH9AqAbvNYAv4c3J8OhAa/LwJy1E2eKz/TacT6sQ411gKQz+cnVi60AGhLQGIKpMlnEEfk\ncDh0IcbZbDaYd0ABWB4UgSUiIiCDQ2cAyvHk5MQ91j4BHfqrHYz+QAYwIQB6ru0P/tDzpa8yHZDX\n1DkOIgB6FaFQKAQHv8QyAGNLoNPpoFgsRmoZ6Pf0i5nI5whNX+R5FIPkoAgsEX86ILEAJycnziFY\nq9XcykC9Xo/k/vd6vZkD2LcEdAMmLQDtxdc/l3NNKpWKfAY/izCfz8cKgF/5SAtAKJU5JAIhQZLH\nJDkoAktEWwKSHyArAnfv3sXJyclEzoBvCcjr6NcMERID+X0tAHKH1Wv8vrjo836/H+u5z+Vyc4mA\nODblNWWFIDS4pe8hsSPLgSKwRPyQXRGCWq3m4gB0boDMsXU47ixkQIWaePpDMQra0RjyOczz3v1+\n331OvcQoKxoAcHp6OhFPIK3b7U4kLYnjsN/vO8fheSohk/NDEVgiemBIdp/cOWWVQO6eviMwROif\nPpTtJxl/qVRqItZAP9bXdM6B3LXjrA7/80ncQ7fbnRicEtbsi2Gz2cTu7u5E//2VBB3GrM8pAMlB\nEVgyIRHQS4W6BoAMzHkDgoCxCEiMv5/ok06ngwVLQgVMdBSi75CLwxc5HVAkz61Wq5E6iTptWlcy\nCh1D6cu+T4EsDkVgieg7pV5e0yXA/FoAoahAXwB0QJCIQLlcRqVSQaVSwd7eHiqVCtLpdHAJzy9A\n0ul03B1cpjDz3mn1FEBCmPV0QlKdtfjJwPaXDv3ml0qX1Qs/S5EsBkVgicwzHfB3FfKnA6GB74tA\nsVhEuVzG/v4+Dg8PcXBwgMPDQ1dUJNR0arJ2IsqAPs/nE+chEBUFYGwJiAD4c/9QjQLdisWis5DE\nApAkJYpAclAElkhIBLQl0Gw2J2oBhKYDIQEIWQIHBwe4cuUKrl27hqtXryKbzbrYA33M5/NoNBqR\nCD1x1kkdgnksAf355LF8Bpla1Go1tFqtYKZi6G6vj+VyObKsKAIgokCSgSKwZPzpgO8TiMswDP2T\nh/IDtAjs7+/j6tWruH79Om7cuIFsNhvZp6Ber7vCIVoA9B1dPPLnmQ7o1/DjFGq12kQZcznm8/lI\noRLfp9HtdiM+ALEe9HIjWRyKwBKZZzrgB+rEeeXjMuxEBCqVCvb393HlyhU88sgjODo6QjabRbVa\nRbFYdPNwEQA/ZFiLlP75PJ9P5uqh6Uq1Wp3a91KpFNtkmiECsLu766YH8yxhkvmgCKyIWeG70+5s\nfkqtHkhiYoujTSoB7e3tuaW0aU3eOyRW04KI4sKPfaQCUgi9/VmoZbNZ95nOGz9B5ocisER0BJwM\nVlkHl7h7Hb0XCtbxBUCO+txPHJJjJpNxAhEK2/Xz+oGoRSLJPaECIuIHSAIdbKSLl/h1FOKyKMli\nUASWjB6MWgDEtJV/cL9QaFy8vH8XD2UQ6gg7EQFfAMRB58//ZZANBgNkMpnYYKM4v8V5mSYAuo6C\nv3RKIUgOisASkUGpa/nppTEdIKSX2eLCdv3Br69pIdDvKyLgC4BsS+7P/fVgzGQyE4FFfrbhoswj\nArQElstMETDGvAzgIwCOrbXvP7t2COCbAB4H8AaAj1trT5fYz0uLLwIS+SYi4IfahgRAD37/sV+i\nK2QJhARgd3c3En6r/QK6zFg2m3WFRQAkOhWYRwD8oqZJWSDkAfNYAl8D8GUAr6prnwfwHWvtfzDG\n/BsAXzi7RhS+JeAHyUikXmipLhS2G3Lq+QM/JAK+AMgg29nZce/rD0ZZ5/djCeRnScbu0yewXmaK\ngLX2e8aYx73LzwD4F2fnrwD476AITCCD1bcEdFhsXLDOrNcMTQf8Jkt9IgC+912mCf7gkyhGsSCk\nfzIwk4zbp09g/VzUJ/CItfYYAKy1t40x1xPs00NFyCegpwPA5DzcN/+B+LDh0DRAWwLTBqzU+vNT\ngEUEZFriJwkllcbr1zqIswJoCSwXOgaXiCT0PPHEE0t7j+9+97sXfm6xWMS1a9cS7M0kXNPffC4q\nAsfGmBvW2mNjzE0Ad5Ls1MNCo9HAu+++O9Hu3r2Ld999FycnJxM7EevHvV4veIeXO//JyQk++MEP\n4tFHH8XR0RGOjo7c+aOPPopKpTIRqquPrVbL7Xso7e7du+5cNkX1d0qWc/EbxCFhxHFkMpmpacOH\nh4e4efMmjo6OgsdyuZz0n2wrmVcEUmdN+DaATwP4IoBPAXgt2W49HGh/gDgEi8WiKyQia/c6bFeH\n9YbW72VgyR3Wz9Ov1WouAUfm7xIU5B8lj0H6I6nF/n4H6/DO+0uhZHnMs0T4dQBPA7hqjHkTwAsA\nfhvAHxljfgPAmwA+tsxOXlbkju3HB5RKJRcopLPuJK/Ar78vyODT6/Sh0mW7u7uREl1+Bp8cdX1A\nqS+gC4zEzclX8b3Nc40kwzyrA5+M+dEvJNyXhw7fEhCHoN7VVwuAlNPyLQF/4GmvvV+wo1aruUQh\nWQYMbT+ey+VcFWRtBcRZAtM2P0nqu5rnGkkeOgaXiFgCejrg32F1erFO8/XNYb00pmMI9PMbjUbk\nNXq9XqR4pz7K3T40HfCFwE9xXqY1wIG/eigCS8b3Cei4e9mnUFKLtSUQWoYLrZH70wFtSXS73Ym6\nfXJNBri2BObxCaxqeS6UOOWfk2SgCCwRP3lIBpX8bDAYuJ2JCoVC0BIApu87oEVAP3c0Gm8oKqW7\nOp1OcCoSNx3Qv+eXC1+XGJDlQBFYIqFEHp0i3Ov1UK/XnSNPF/0Qa0Cb/yEx0AVM/S3EZeMTCVH2\nz4fDodsBSTY+8XdA8q0C3Y9FB6gf6KS3LhcHplzztzInyUERWCKhsGFdGkvCh3WdfbEGZCDoO69e\nJhRkeVEsAp0ZqKsFhdbih8MhGo3GRNNioMuha6sgKUIJVrKKIt+P/k7iIirJxaEILBn9T66X2FKp\nlPtH1yIgXnxdB9AvNqIdg7rct183UK86+Dv85HI5DIdDN+B1GXQ51z4CXdUnqemAL5JxVYjFQtLf\nCUUgOSgCS8TPItQCIME6+m4XZwnoJB4/gUgHGvnbiHc6nYio+G00GkVWBvRRpgQ6lyBpS8CfLvmp\n1jJNku9FO01JclAEloz8k2sBkGvTLAERAT93X8cIAA8sgVC6r34dHS0oTZyHcS1uu7IkHYPTUq19\nS2Daygm5OBSBJaIdXzJX1//0YglM8wloQgVH5K4v5yIA8vxQuW9pMmXw9yfUG6H4qbzSkvp+pvkE\nRARCPgGSHBSBJSL/5AAiHnCZV8uavZ4O+D4BIZQ3AMBZCqF0Xz/hSHviZSD5gzyUvx8qhJqUT2Ba\n+TV/1YSWwHKgCCwZvcefnhZoEZhlCfgCoAeADEpZcoyrPiSEIhGnlRL3z/Uxqe/HD63WPgEtkPQJ\nLAeKwBKZlQnnFxnRu/HIFlxS5FOKfoqg6K2/4gblooN1WtTetLyG87y+tga0RRDyZ/j1FEkyUATW\niBTylBTjSqWCg4MDdDod9Pt95PP5yMah0oAHxTr0XXEZd+pQ2TK57r+n38jlgCKwRiTNWPbkq1Qq\nTgBGo5HbOFSa+Ai0M1DHBchRxxEsir5bh3wKIb+B7g/ZfCgCa0QsAS0CIgCyYWe1WkUul3NecREA\nsQjE0ahZhgD4tQv9JUnpm1yjuX55oAisEbEEZDqgBUDn/msLQIcHy2sAk8uHSQqBH9uvlxhlwGsf\nBQXgckERWCPaEiiVShEBkNUC3wLodDpoNpsTIqDvvsvwCejBL446AJFgJvoDLicUgTWiB7zcQbWP\nIJ/POwEQC6DZbLotxOQ1NEluFBrK8tMee/kdIRTMRDYfisAakUjCfD7vzmVq0Ov1XKafrhlQr9ed\nhSCvIQLitySTfELLeNoZKXEMTPe9fFAE1oguOuIXJRX/QLPZjG3AeG8DP7z3vJWA4oKK/HV7XahU\nREAyDeWoqyfPIrTxiA5bnlbdiFOO5KAIrBn/TiuRhQCc07BUKmFvb88tH+r5/9HR0UQx0Hmy/eKs\nB920COhCHzq5SdKOJQ1ZT1/mQYc7S+KSxEZIHoFUQdbbkpHkoAhsAGLSawFIpVIujr5UKrmBIAIg\nc/JHH300uG2XWBIa/3Hc+n9cpR+/9Xo9V5lIkpVEAGQJcxpiBYTKrutISr+wCXcmThaKwJrRloBO\nNx4MBtjZ2XH7FGgLQOLsgbEIhNJ95fdnvbde9w+dz9q8RJyUYgFIGfN54vt9EZBpRbvddg5SXepM\npzKT5KAIrJGQ+a0fa0tAC4DcIYHxdMAXAV0qfNb7+2v/cS0UJyCDVU8B9LV50NMBsQRkyiEioCsc\n0RJIHorAmvEtAf14OByiUChMWAC7u7tuHz5tCfi1AUIioAePrEj4d3k59wXAPzYajQkBaDQac4tA\nyBLY2dlBu912EZMiAr5PgCKQHBSBDUD7BCRgSP7JQxZAuVxGp9MBMBYBGSC6FqDeWnza+/rZen6l\n39CGqNLq9boTANn3QKYHF5kOiCUgvgk9HfB9AiQ5KAJrRi/PhRx3WgD0AJcluKOjowkRkDavCMQt\nAer03VCBkmq16rz59Xod1WoVhULhXNOB0Wjk+u4XDKElsBooAmvEHyhxA0eEQEqWSwOAg4ODCQtg\nXhGQGAU/FkCaH/yjBUCmAaVSyRX/kNeat/KPX0bd36A1FCdAKyB5KAIbju889M1sXYfQjzlYZDoQ\nKuIxa2AvGiXIKMP1QBG4BMQJAPBABEKhvfOIQNzyn04X9iMKVzFYV/le2w5FYIOZZgHI4NBVjNPp\nNAaDgbMC5hGBacuA66rlFxr4FIPlMVMEjDEvA/gIgGNr7fvPrr0A4FkAd85+7Xlr7Z8urZdbjgwA\nfWcWx5hfytyvFDzrdeMChULz+vPkIZwXDvz1MY8l8DUAXwbwqnf9RWvti8l3iWj8gaCXEwFEtirz\ny3zNIwLTvP9abHTZsmV65jnwV89MEbDWfs8Y83jgR/xrrQA/s88vJqq3Nztvsc9ZCUSh1xSmvXYS\nA9n/3Em+NomyiE/gOWPMrwP4GwC/Za09TahPxMMflBq9OjDvIPVfO+7cX8I7z2snNVg56JfPRUXg\nKwD+nbV2ZIz59wBeBPCvk+sWAeabJ+tdilZNuVzGY489hqeeeir2d7iuv/lcSASste+ohy8B+JNk\nukM2Ce1bCPkbTk5O8Oabb+Ktt97CW2+9NXF+fHyMQqEQnKZI2rTswBQ6Hhwc4OjoCDdv3nRNP5b8\nCbIY864BpaB8AMaYm+pnHwXwwyQ7RTYTvXavly5Dqcc7OzsAMBF3ME/QEVkt8ywRfh3A0wCuGmPe\nBPACgJ83xjwJYAjgDQCfWWIfyRrxcxr0Yz9CMVSIVCoQ6WkB9yXYLOZZHfhk4PLXltAXsqGEhECO\nsywBqZHAfQk2F0YMkrnQA9cXhGklycUSkOexLPnmQREg5yZkCfibk2ifAABXJ4FTgc2DIkDOje8g\nnGUJaAuAzsHNgyJA5iI0HdCDX/ZLkOU9qYFYKBTcwBcxSHKXJLI4FAEyE39FQF+XuodS+qxYLKJc\nLmN/fx/AOKCo3W47B6GIAC2BzYEiQKbiC4D27ks9AikPLuXRK5UKWq0WgLEI+ALQ7/eD5dTIeqAI\nkLnQAiBHPRWQKUCpVEKr1XKbj5TL5YgASDFRisDmQBEgMwkJAPDAEtDTAb1bEjDeK9EXALEMyGZA\nESBz4QuALPmJJaB9AlIeHBhbAloApIoxRWBzoAiQudECIMt8viXQ7XYjYcLlcjmyxyBFYPOgCJBz\nM80xKDsfyXxfNkpptVrI5XJzb0xCVgdFgMQyT4lxPSUQMZA9AwG41OBcLjexoQnZDCjJ5MLoiMFQ\nCDGAiTTi8woAVxCWD0WALIyfTizhw8CDQqj+bkaziCuVRlFIHooAWYhQ/kCcJaCtgDgh4CBfPRQB\nshChKkNxloBfyVgTGvwUhNVAESAXJi6lWEqKAWFLYNZ0gIKwWigCZCHiag1OswTmwd9fgSwPigBZ\nCH91wPcJ+IVGp4kBB/x6YJwASQR/9yK9b+IsX4C/yUmo+eXP591hicyGIkDWij/4B4OBSzfu9/vu\nsYQi+6LALcoWhyJA1s4sIYgTgzhLgGJwPigCZG1MG/zpdHoua0AnM7Gc+cWgCJC1EhKCeQUgNB0I\nlUEj06EIkLUSEgG5s19kOsDBf34oAmStxAmAiIAe/L41ICXMNRSB80MRIGtDBwSJGOhKxL4VEPIH\naCHgdOBiUATIWtGDWSwAqUXgWwEhIZDXkCMH//mhCJC14gcK6UEcGvy+EKTT6QmLgkJwPigCZO3o\nTUpDIhAa/NIEEQOGHp8figBZG34IsB8a7FsCcX4Bf/DTIjgfM0XAGHMLwKsAbgIYAHjJWvu7xphD\nAN8E8DiANwB83Fp7usS+kocUf8tyvVlJ3DQg9Pu0BC7GPFmEfQCfs9b+NICfBfCcMeZ9AD4P4DvW\nWgPgLwB8YXndJA8roTgBP4Iwbipw3oQiEmamCFhrb1trv392XgfwYwC3ADwD4JWzX3sFwK8sq5Pk\n4cafDoRWBUJCEJpOUAzOz7nqCRhj3gvgSQB/BeCGtfYYGAsFgOuJ94489MSlEc+zMhBaLiTnZ27H\noDGmDOBbAD5rra0bY/itbzmy10ClUon9nWeffXaFPSIXYS4RMMZkMRaA37fWvnZ2+dgYc8Nae2yM\nuQngzrI6STaTXq+Hdrsd2z784Q/jpZdewu3bt3F8fIzbt2+7dnx8jEajMfX1Dw4OcOvWLdy6dQuP\nPfbYxPn+/n6kpqEudy6VjMhs5v2WvgrgR9baL6lr3wbw6bPzTwF4zX8SIWTzmWeJ8CkAvwbgB8aY\nvwMwAvA8gC8C+ENjzG8AeBPAx5bZUULIcpgpAtbavwSQifnxLyTbHULIquGkiZAthyJAyJZDESBk\ny6EIkI0llBSkz/3H5GJQBMhamHfgnmdzEoYNXwyKAFkJcQPzvEIwSwDI+aEIkLUzbfCGag3E5RHQ\nGrgYFAGyEcRtRx4qRhpKKOLgvzgUAbIyZg3QaVOG8/gDKATngyJANgp/AIfKjjGdOFkoAmSlzOsI\n9B9Pq0BES2AxWGiULITeMSidTruWyYzTTfQ1afo505BB3+/30ev10Ol00G630Wq10Gg0sLOzg1wu\n5wRBSKfTM7ctZxHSB1AEyIXRg9/P55cNRHZ2dpDNZl3Tvyf5/n5QkDAcDt3gb7VaqNfrOD09RT6f\nRzabRa/Xw+7u7kQDENmVyD9SAKJQBMhC+FZANpuN7AmQzWadEPgFP3wRCEUI9vt9dLtdNJtN1Go1\n5HI5ZLNZpFIpdLtdlEollMtl9Ho9956ZTAY7OzsTVsc0y2CboQiQhdACIINci8A0SyCTycTuGyDH\nfr+PTqeDZrPpLAAx97vdLjqdDvr9vnu/dDqNnZ0d5PN5ZDIZ1z85kkkoAuTC+P6ATCaD4XCIbDbr\nBvKs6YC/54BG/AEiAmIBjEYj9Ho9dLvdiAWgBaDX6znBkL7K79AKiEIRIAvhC4B/V581HRABkLs7\nELYEZECLALTbbXS73YgAZLNZ5PN5FItFt5ux9CfO6iAUAbIg/nTAH2SzpgMyiLUQ6B2FRARSqZRz\nFMoKQbfbBRC1AIrFopsiiHNS+qmP5AEUAXJhQtMBfR3AVAHw5+h6e3J5LCLgTw1yuRy63a5zAooA\nlMtldDod9Hq9CRHQ4kIeQBEgCzNtR+BcLudaoVBwbXd31w1UuWtLG41Gkf0I5VyEQByCqVQqsjRY\nLBYjj0ejkdsbQTcA7kgoAmRBfGsAwMR0QO7SpVIJlUoF+/v7aLVaSKfTEQefHP0wYX+LMkGmBo1G\nA9VqFblczlkjg8EAe3t7EdHRIlQoFFb7RW0wFAGSCL4QiEkvFsDu7q4TgVarhU6ng3Q67aIA5THw\nYBogYuJPE4Rut+uiB0UAUqkUBoMBut0uGo2GiyMol8vo9/sA4ISCjKEIkAsTsgLEwSfkcjnk8/mI\nCIhnP51Ou6U/vVLQ7/fdgI9bQtSrBI1Gww1siR+Q6/v7+85RKKsEuVxuRd/Q5YAiQBZC36H1dECu\n7ezsRCyBTqeDbreLwWDglvV8C0Du6KHpgLy+Hux6CtDr9Zx10Gg00Ol03BRCLyGSB1AEyMJoM933\nC4glIEt33W7XRfj5FoR4/0UE5HVk0GthSKVSbsDL87UFUK1W0Ww2nQBkMhnXj16vt7Lv5jJAESAL\nodff/UrZ7a1AAAAHOElEQVRAQNQSEAGQn+mlQEkU0nN7QfsEJF5A3k8/X/sHdnZ20Gw2AUQFoFKp\nON8AGUMRIAvjC4FG+wR0jH8oFqDdbrvgIm0JyNF3DEqOggiAH4fQarWQzWaRy+WcALTbbVoCHhQB\ncmFC0Xf+NQkblnh+iQMAxnN4SRMuFAruDu5bAoIvMP6qgb+CwDqE80ERIEtFZxfmcjn0ej30+330\n+30UCgXk83k3+P28glkhvuLpl+frYy6Xw8HBAa5fv47Dw0Ps7e2hVCqhUCgwUMiDIkCWSiqVciLg\nVwLqdDpThSBOBOS6iIAOBpJowUKhgIODA1y7dg1XrlyJiEA2y397zcxvwxhzC8CrAG4CGAD4PWvt\nl40xLwB4FsCds1993lr7p0vrKbmUaEtAREBM81kCMCvpR0Rgd3cX5XLZBQbJ8eDgAFevXo1YAvl8\nnpaAxzyS2AfwOWvt940xZQB/a4z5b2c/e9Fa++LyukcuO74I6KAfEYFpU4JpSPKQxCDs7+9jb2/P\nHQ8ODnBwcDAxHaAlEGXmt2GtvQ3g9tl53RjzYwDvOfsx8zLJVCQWQCoO6eVBmQ7MsgTi8C2B/f19\nHB4e4sqVKzg8PHRiUKlUUKlUUCwW6RMIcC5JNMa8F8CTAP4awIcAPGeM+XUAfwPgt6y1p4n3kFxq\ndMERGXwiDP50YF6fgH4N8QmUy2Xs7e3hypUruHbtGq5fv479/X0Ui8VI43RgkrmLrp1NBb4F4LPW\n2jqArwD4KWvtkxhbCpwWkAlCPgGJGxAHnghByBLwm145iLMErl+/jhs3buDGjRu4du0apwMzSM2z\nZmqMyQL4LwD+q7X2S4GfPw7gT6y175/xUlygJWR9BE2reSXxqwB+pAXAGHPzzF8AAB8F8MPF+kce\nRnTBEB28MxgMUKvVcOfOHdy5cwfHx8eR4507d9BoNADErw6USiVn+uujnJfL5UiNQ32ez+dX+TVs\nNPMsET4F4NcA/MAY83cY382fB/BJY8yTAIYA3gDwmSX2k1xi4sp9S16BePYlpl8GqSQHxVEsFnF4\neOicgQcHB84BKHN/P5SYm49MMs/qwF8CCFVhYEwAmUloTi/XZXmvXC5H0n0l1r/dbkdexyefz7sV\nAGnTRGDeSMRtgx4SsnRCpcdEBMSzD0xm+81K9MnlciiVShOtWCy6FQdxTPp7IZIHUATIUvFrDegi\noiICwIM1fxGAg4ODYMqvfr1sNhupGZjP5yPnOzs7ExulcjowCUWALBV//7/QvoNaAEI7C8Whlx51\nnIGcax9A3J6EZM4lwgThEuGW4f9/+XsOSpqvRBP6j6fhxw3456Hcgy3fhCT4oSkChGwPQRHgNq2E\nbDkUAUK2HIoAIVsORYCQLYciQMiWQxEgZMuhCBCy5VAECNlyKAKEbDkUAUK2HIoAIVvOqrMItzJr\ng5BNhpYAIVsORYCQLYciQMiWQxEgZMuhCBCy5VAECNly1lJo1BjzSwB+B2MRetla+8V19CMOY8wb\nAE4x3lilZ639wJr78zKAjwA4lq3ejDGHAL4J4HGMN3/5+Lo2hI3p3wsAngVw5+zXnrfWrmWvCmPM\nLQCvArgJYADgJWvt727Kdxjo3+9Za7+8qu9w5ZaAMSYN4D8C+FcA/gmAXzXGvG/V/ZjBEMDT1tp/\num4BOONrGH9fms8D+I611gD4CwBfWHmvHhDqHwC8aK39Z2dtnZvV9AF8zlr70wB+FuPdtN+HzfkO\n/f79phoTS/8O1zEd+ACAv7fW/oO1tgfgGwCeWUM/ppHCBk2VrLXfA3Dfu/wMgFfOzl8B8Csr7ZQi\npn/AhgSHWWtvW2u/f3ZeB/BjALewId9hTP/ec/bjpX+H6/hHfw+At9Tjt/HgA28KIwB/Zox53Rjz\n7Lo7E8Mj1tpjYPxPBOD6mvsT4jljzPeNMf/JGLO/7s4AgDHmvQCeBPBXAG5s2neo+vfXZ5eW/h2u\nQwRCyrZppch/zlr7MwB+GeM/wofW3aFLyFcA/JS19kkAtwG8uOb+wBhTBvAtAJ89u+Nu1P9doH8r\n+Q7XIQJvA/gJ9fgWgH9cQz9ikS3XrbXvAPhjjKcwm8axMeYGMN4mHg+cRxuBtfYda60MspcA/PN1\n9scYk8V4gP2+tfa1s8sb8x2G+req73AdIvA6gCeMMY8bY3IAPgHg22voRxBjTPFMkWGMKQH4RQA/\nXG+vAIwtKG1FfRvAp8/OPwXgNf8JKybSv7NBJXwU6/8OvwrgR9baL6lrm/QdTvRvVd/hqncgAuCW\nCL+EB0uEv73yTsRgjPlJjO/+I4yXUP9g3f0zxnwdwNMArgI4BvACgP8M4I8APAbgTQAfs9aebFD/\nfh7jue0Q4+W3z8j8ew39ewrA/wDwA4z/riMAzwP4nwD+EGv+Dqf075NYwXe4FhEghGwOG7MMRghZ\nDxQBQrYcigAhWw5FgJAthyJAyJZDESBky6EIELLlUAQI2XL+P8+9eOBGMz3ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2a68d94c10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the image\n",
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(sample_img).set_cmap('Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Labels\n",
    "\n",
    "We also need the labels for each image to train the model.\n",
    "\n",
    "![mnist.train.ys](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.21.20%20AM.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(55000, 10)"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check MNIST labels shape\n",
    "mnist.train.labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.])"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show MNIST label data\n",
    "sample_label = mnist.train.labels[5]\n",
    "sample_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single neuron recognizes single digit\n",
    "\n",
    "A neuron takes 784 inputs, mutiplies them with weights, and checks if the sum exceeds a threthold. \n",
    "\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%206.08.38%20PM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a neural network as a graph:\n",
    "\n",
    "One neuron for each digit, total 10 neurons.\n",
    "\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.22.26%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Or, as a matrix operation:\n",
    "\n",
    "It's much easier to use matrix operation to write this!\n",
    "\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.22.40%20AM.png)\n",
    "\n",
    "<br/>\n",
    "$${\\Huge y=softmax(Wx + b)}$$\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Or, as a TensorFlow graph:\n",
    "\n",
    "In TensorFlow, you'll define a \"graph\" of the calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a neural network (softmax logistic regression)\n",
    "import tensorflow as tf\n",
    "x = tf.placeholder(tf.float32, [None, 784]) # a placeholder for inputting the image \n",
    "W = tf.Variable(tf.zeros([784, 10])) # weights\n",
    "b = tf.Variable(tf.zeros([10])) # biases\n",
    "y = tf.nn.softmax(tf.matmul(x, W) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The trained weights will look like this:\n",
    "After a training, the weights will look like this. The blue area has **positive** weights, and the red area has **negative** weights.\n",
    "\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.22.14%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Softmax?\n",
    "\n",
    "Softmax converts an array of values into an array of **probabilities** (0 - 1.0).\n",
    "\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "$${\\Huge softmax(n) = \\frac{\\exp n_i}{\\sum \\exp n_i}}$$\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Container object of 10 artists>"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEDCAYAAAA4FgP0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE0VJREFUeJzt3X+wXGV9x/H35VLa4k2R6BIwSGIC+WLpIOCYoTi2VEXQ\nxgTrL6Bj+eGoU6XSOoMj0Bl0pjOITltQnGmhkQHGFKeM8msUgTJth2nRqNBige9NDYHEQIi910hM\nh8bL7R+7mNvrbu5Nzrm75+Z5v/7JnrPP3e+zz272s885Z88ZmpycRJJUroMG3QFJ0mAZBJJUOINA\nkgpnEEhS4QwCSSqcQSBJhTu4jgeJiLXAKmBbZp7Y5f7fBe4ANnZWfS0z/6KO2pKkamoJAuBG4IvA\nzXtp8y+ZubqmepKkmtSyaSgzHwTGZ2g2VEctSVK96poRzMapEfEwsBW4NDMf62NtSVIP/dpZ/D1g\nSWaeDFwH3N6nupKkGQzVda6hiFgC3NVtZ3GXtk8Cr8/Msb21m5ycnBwacouSJO2Dff7QrHPT0FCv\nDkTEoszc1rm9EhiaKQQAhoaG2L79+Rq7OH+1WgscCxyHqRyLPRyLPVqtBfv8N3UdProOOB14RUQ8\nDVwJHAJMZub1wHsi4o+B3cD/AO+vo64kqbpagiAzz5vh/i8BX6qjliSpXv6yWJIKZxBIUuH6+TsC\n7aeJiQlGR0cZG9s557WWLl3G8PDwnNeR1BwGwTywadNGLvn8nRx62BFzWmfXjue49tLVLF9+3JzW\nkdQsBsE8cehhRzBy+OJBd0PSAch9BJJUOINAkgpnEEhS4QwCSSqcQSBJhTMIJKlwBoEkFc4gkKTC\nGQSSVDiDQJIKZxBIUuEMAkkqnEEgSYUzCCSpcHVdvH4tsArYlpkn9mjzBeDtwM+ACzLzkTpqS5Kq\nqWtGcCNwZq87I+LtwPLMPA74CPA3NdWVJFVUSxBk5oPA+F6arAFu7rT9NnBYRCyqo7YkqZp+7SNY\nDGyesvyjzjpJ0oD1KwiGuqyb7FNtSdJe9OuaxVuAV09ZPhrYOps/bLUWzEmH5pPx8ZG+1Vq4cKTx\nY970/vWTY7GHY7H/6gyCIbp/8we4E/gY8NWIOBX4SWZum82Dbt/+fE3dm7/Gxnb2tVaTx7zVWtDo\n/vWTY7GHY7HH/gRiXYePrgNOB14REU8DVwKHAJOZeX1mfiMi3hER/0X78NEL66grSaquliDIzPNm\n0ebiOmpJkurlL4slqXAGgSQVziCQpMIZBJJUOINAkgpnEEhS4QwCSSqcQSBJhTMIJKlw/Trp3Lw0\nMTHBpk0b+1Zv6dJlDA8P962eJIFBsFebNm3kks/fyaGHHTHntXbteI5rL13N8uXHzXktSZrKIJjB\noYcdwcjhXkNH0oHLfQSSVDiDQJIKZxBIUuEMAkkqnEEgSYUzCCSpcAaBJBWurovXnwVcQztY1mbm\n1dPuPx/4PLCls+q6zPxyHbUlSdVUDoKIOAi4DngLsBVYHxF3ZOYT05rempkfr1pPklSvOjYNrQQ2\nZOZTmbkbuBVY06XdUA21JEk1q2PT0GJg85TlLbTDYbo/iIg3AaPAJzJzS5c2Uk8TExOMjo4yNraz\nL/U8CaBKUUcQdPumPzlt+U5gXWbujoiPADfR3pQ0o1ZrQcXu7b/x8ZG+1lu4cKTr8+1nP3r1oQlG\nR0f5wGXr+nYSwFuuOo8VK1bMea0qmvpaDYJjsf/qCIItwDFTlo+mva/gFzJzfMriDcD/25m8N9u3\nP1+pc1X065vn1Hrdnm8/+9GrD00wNrazrycBbPJYQPuDr8n96yfHYo/9CcQ69hGsB46NiCURcQhw\nDu0ZwC9ExJFTFtcAj9VQV5JUg8ozgsyciIiLgXvZc/jo4xHxGWB9Zt4NfDwiVgO7gTHggqp1JUn1\nqOV3BJl5DxDT1l055fblwOV11JIk1csL00j7yEuY6kBjEEj7yEuY6kBjEEj7wUuY6kDiSeckqXAG\ngSQVziCQpMIZBJJUOINAkgpnEEhS4QwCSSqcQSBJhTMIJKlwBoEkFc4gkKTCGQSSVDiDQJIK59lH\npXloYmKC0dHRvl3P2msiHNgMAmke8poIqpNBIM1TXhNBdaklCCLiLOAa9ly8/upp9x8C3Ay8Hvgx\n8P7MfLqO2pKkairvLI6Ig4DrgDOBE4BzI+L4ac0+CIxl5nG0A+NzVetKkupRx1FDK4ENmflUZu4G\nbgXWTGuzBripc/s24C011JUk1aCOTUOLgc1TlrfQDoeubTJzIiJ+EhELM3Nsbw/cr6Mi9nZExK4d\nz815/dnU6Uc/9lZjYmKCTZs2znkfXtLrNSnp9ZipThP60IT3RVOOoGrCWOyvOoJgqMu6yRnaDHVp\n80s+cNm6OT8qYteO57jlqvNYsWLFL923cOHruOWqkTmtP9Xy5cu7vrD97EevPoyOjvb1KJVur0mJ\nr0evfjShD9CM98Xo6GhfPiv21oeX+jHosdhfdQTBFuCYKctHA1untdkMvBrYGhHDwG9k5vhMD9yv\noyLGxnayffvzXe87/PCj5rz+nn7s6nnfihUrevaxH30YG9vZ16NUer0m/RqHdh96vx5NeF80YSya\n8L5oQh+a1I9Wa8E+P1YdQbAeODYilgDPAOcA505rcxdwPvBt4L3AAzXUlSTVoPLO4sycAC4G7gX+\nE7g1Mx+PiM9ExKpOs7XAKyNiA/CnwKeq1pUk1aOW3xFk5j1ATFt35ZTbLwDvq6OWJKlennROkgpn\nEEhS4QwCSSqcQSBJhTMIJKlwBoEkFc4gkKTCNfrCNIM+0ZoklaDRQXDLVef17eyjklSqRgdBP0+q\nJUmlch+BJBXOIJCkwhkEklQ4g0CSCmcQSFLhDAJJKpxBIEmFMwgkqXAGgSQVrtIviyPicOCrwBJg\nE/C+zNzRpd0E8O/AEPBUZp5dpa4kqT5VZwSfAu7PzAAeAC7r0e5nmXlKZp5sCEhSs1QNgjXATZ3b\nNwG9PuSHKtaRJM2RqkFwRGZuA8jMZ4FWj3a/GhHfiYh/jYg1FWtKkmo04z6CiLgPWDRl1RAwCfz5\nPtQ5JjOfjYjXAA9ExH9k5pOz+cNWa8E+lDmwDXIsxsdH+lpv4cKRns/X98Qegx6LJrwvmtCHJvVj\nf8wYBJl5Rq/7ImJbRCzKzG0RcSTQ9SovndkCmflkRPwTcDIwqyDwNNRtrdaCgY5FP64LMb1et+c7\n6HFokiaMRRPeF03oQ5P6sT/hUHXT0J3ABZ3b5wN3TG8QES+PiEM6t18JnAY8VrGuJKkmVYPgauCM\niEjgrcBnASLi9RFxfafNa4HvRsTDwD8CV2XmExXrSpJqUul3BJk5RjsApq//HvDhzu1/A06sUkeS\nNHf8ZbEkFc4gkKTCNfri9ZI0n+za0fXAycbXMQgkqQZLly7j2ktX97VeXQwCSfNeE76JDw8Ps3z5\ncX3pR90MAknz2tKly7jlqvP69oOuOr+JN4VBIGleGx4eZsWKFQP/lfV85lFDklQ4g0CSCmcQSFLh\nDAJJKpxBIEmFMwgkqXAGgSQVziCQpMIZBJJUOINAkgpnEEhS4QwCSSpcpZPORcR7gE/TvkD9GzLz\n+z3anQVcQzt41mbm1VXqSpLqU3VG8CjwLuCfezWIiIOA64AzgROAcyPi+Ip1JUk1qTQjyMwEiIih\nvTRbCWzIzKc6bW8F1gBPVKktSapHP/YRLAY2T1ne0lknSWqAGWcEEXEfsGjKqiFgErgiM++aRY1u\ns4XJ2XUPWq0Fs216wBvkWIyPj/S13sKFIz2fr++JPQY9Fr4vDgwzBkFmnlGxxhbgmCnLRwNbZ/vH\nXnWordVaMNCx6NdlAKfW6/Z8Bz0OTdKEsfB90Tz7E4h1Xqqy136C9cCxEbEEeAY4Bzi3xrrqkyZc\nIFxS/aoePno28EXglcDdEfFIZr49Io4CbsjMVZk5EREXA/ey5/DRxyv3XH21dOkyrr10dV/rSeqP\nqkcN3Q7c3mX9M8CqKcv3AFGllgZreHiY5cuPG3Q3JM0Bf1ksSYUzCCSpcAaBJBXOIJCkwhkEklQ4\ng0CSCmcQSFLhDAJJKpxBIEmFMwgkqXAGgSQVziCQpMIZBJJUOINAkgpnEEhS4QwCSSqcQSBJhTMI\nJKlwBoEkFa7qxevfA3waeC3whsz8fo92m4AdwIvA7sxcWaWuJKk+lYIAeBR4F/C3M7R7ETg9M8cr\n1pMk1axSEGRmAkTE0AxNh3AzlCQ1Ur8+nCeBb0XE+oj4UJ9qSpJmYcYZQUTcByyasmqI9gf7FZl5\n1yzrnJaZz0ZEC7gvIh7PzAdn84et1oJZljjwORZtjsMegx6L8fGRvtZbuHCk53Me9FjMZzMGQWae\nUbVIZj7b+Xd7RHwdWAnMKgi2b3++avkDQqu1wLHAcZiqCWMxNraz7/W6PecmjEVT7E8g1rlpqOt+\ngog4NCJGOrdfBrwN+EGNdSVJFVQKgog4OyI2A6cCd0fENzvrj4qIuzvNFgEPRsTDwEPAXZl5b5W6\nkqT6VD1q6Hbg9i7rnwFWdW4/CZxUpY4kae54SKckFc4gkKTCGQSSVDiDQJIKZxBIUuEMAkkqnEEg\nSYUzCCSpcAaBJBXOIJCkwhkEklQ4g0CSCmcQSFLhDAJJKpxBIEmFMwgkqXAGgSQVziCQpMIZBJJU\nuErXLI6IzwHvBF4AfghcmJk/7dLuLOAa2sGzNjOvrlJXklSfqjOCe4ETMvMkYANw2fQGEXEQcB1w\nJnACcG5EHF+xriSpJpVmBJl5/5TFh4B3d2m2EtiQmU8BRMStwBrgiSq1JUn1qHMfwUXAN7usXwxs\nnrK8pbNOktQAM84IIuI+YNGUVUPAJHBFZt7VaXMFsDsz13V5iKEu6yZn28FWa8Fsmx7wHIs2x2GP\nQY/F+PhIX+stXDjS8zkPeizmsxmDIDPP2Nv9EXE+8A7gzT2abAGOmbJ8NLB1th3cvv352TY9oLVa\nCxwLHIepmjAWY2M7+16v23Nuwlg0xf4EYtWjhs4CPgn8Tma+0KPZeuDYiFgCPAOcA5xbpa4kqT6V\nggD4InAIcF9EADyUmR+NiKOAGzJzVWZORMTFtI8weunw0ccr1pXUELt2PHdA1SnR0OTkrDfXD8Kk\n0702p75tjsMeTRiLiYkJNm3a2Ld6S5cuY3h4+JfWN2EsmqLVWtBtv+xeVZ0RSCrY8PAwy5cfN+hu\nqCJPMSFJhTMIJKlwBoEkFc4gkKTCGQSSVDiDQJIKZxBIUuEMAkkqnEEgSYUzCCSpcAaBJBXOIJCk\nwhkEklQ4g0CSCmcQSFLhDAJJKpxBIEmFq3rx+s8B7wReAH4IXJiZP+3SbhOwA3gR2J2ZK6vUlSTV\np+qM4F7ghMw8CdgAXNaj3YvA6Zl5siEgSc1SaUaQmfdPWXwIeHePpkO4GUqSGqnOD+eLgG/2uG8S\n+FZErI+ID9VYU5JU0Ywzgoi4D1g0ZdUQ7Q/2KzLzrk6bK2hv+1/X42FOy8xnI6IF3BcRj2fmgxX7\nLkmqwdDk5GSlB4iI84EPA2/OzBdm0f5K4PnM/KtKhSVJtai0aSgizgI+CazuFQIRcWhEjHRuvwx4\nG/CDKnUlSfWpNCOIiA3AIcB/d1Y9lJkfjYijgBsyc1VEvAb4Ou3NSQcDX8nMz1bstySpJpU3DUmS\n5jcP6ZSkwhkEklQ4g0CSClfpl8VzpXM00jW0g2ptZl494C4NREQcDdwMHAlM0N4B/4XB9mqwIuIg\n4LvAlsxcPej+DEpEHAb8HfBbtE/hclFmfnuwvRqMiPgz4IO0x+FR2uc8+9/B9qo/ImItsArYlpkn\ndtYdDnwVWAJsAt6XmTv29jiNmxF0/qNfB5wJnACcGxHHD7ZXA/Nz4BOZ+ZvAbwMfK3gsXnIJ8Nig\nO9EA1wLfyMzXAq8DHh9wfwYiIl4F/AlwSueD8GDgnMH2qq9upP1ZOdWngPszM4AH6H0OuF9oXBAA\nK4ENmflUZu4GbgXWDLhPA5GZz2bmI53bO2n/Z1882F4NTmeG9A7a34SLFRELgDdl5o0Amfnzbmf9\nLcgw8LKIOBg4FNg64P70TecMDePTVq8Bburcvgk4e6bHaWIQLAY2T1neQsEffi+JiKXASUCR0/+O\nvwYupf2blJItA34cETdGxPcj4vqI+PVBd2oQMnMr8JfA08CPgJ9MOxlmiY7IzG3Q/jIJtGb6gyYG\nwVCXdUX/x+/8Mvs24JLOzKA4EfH7tLeDPkL7PdLtfVKKg4FTgC9l5inALtqbA4oTES+n/Q14CfAq\nYCQizhtsr+afJgbBFuCYKctHU9BUb7rOdPc24JbMvGPQ/RmgNwKrI2Ij8PfA70XEzQPu06BsATZn\n5nc7y7fRDoYSvRXYmJljmTkBfA04bcB9GrRtEbEIICKOBJ6b6Q+aGATrgWMjYklEHEJ7x8+dA+7T\nIH0ZeCwzrx10RwYpMy/PzGMycxnt98QDmflHg+7XIHSm/ZsjYkVn1Vsodwf608CpEfFrETFEeyxK\n23E+fYZ8J3BB5/b5wIxfIBt3+GhmTkTExbSvfvbS4aOlvbAARMQbgT8EHo2Ih2lvIrs8M+8ZbM/U\nAB8HvhIRvwJsBC4ccH8GIjO/ExG3AQ8Duzv/Xj/YXvVPRKwDTgdeERFPA1cCnwX+ISIuoh2U753p\ncTzXkCQVrombhiRJfWQQSFLhDAJJKpxBIEmFMwgkqXAGgSQVziCQpMIZBJJUuP8DVUoGcvpUgBcA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2a68e9f410>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "i = np.arange(0, 10)\n",
    "n = np.random.randn(10)\n",
    "plt.bar(i, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEDCAYAAAA849PJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFGlJREFUeJzt3XGQXWdZx/HvJqVo2Rgau9UxIQltykPBqbbVqiiItEyD\nxoY/gCaCBosjTokWUMZWB2N0GAwdh9apTEUjUAcMUDs0MFIzUGCGGWhTWgTb9EkwbJolKQSzrakd\nbXq7/nFP2ctlm3t2c/fek32/n396zznve8+zb9v93Xvec94dmZqaQpJUnkXDLkCSNBwGgCQVygCQ\npEIZAJJUKANAkgplAEhSoU6r0ygi1gI30A6M7Zm5rev4m4G3AC3gGPC7mflgRKwC9gAPVk2/nJlX\n96t4SdLc9QyAiFgE3ARcChwCdkfE7Zn5YEezD2fm31Xtfx14L/Cq6tg3MvOi/pYtSTpZdS4BXQLs\ny8wDmXkc2AGs72yQmY91bI4CT3Vsj5x0lZKkvqtzCWg5cLBje4J2KHyfiLgaeDvwLOAVHYdWR8RX\ngP8G3pmZX5x7uZKkfqnzDWCmT/A/sH5EZr4vM9cAfwy8s9p9GFiZmRcDfwh8JCJG51qsJKl/6nwD\nmABWdmyvoD0X8Ew+CtwMkJlPAE9Ur++NiP8EXgDc+0ydp6ampkZGvGokSbM061+cdQJgN7CmuqPn\nMLAB2NjZICLWZOY3qs11wN5q/1nA0cx8KiLOAdYA+090spGREY4cOTa7n2KBGhtb4lhUHItpjsU0\nx2La2NiSWffpGQCZ2YqIzcAupm8D3RMRW4HdmfkpYHNEXEb70/4ksKnq/jLgLyLiOO1bRN+cmY/M\nukpJUt+NNHA56CkTvc1PN9Mci2mOxTTHYtrY2JJZXwLySWBJKpQBIEmFMgAkqVAGgCQVygCQpEIZ\nAJJUKANAkgplAEhSoQwASSqUASBJhTIAJKlQBoAkFcoAkKRCGQCSVCgDQJIKZQBIUqEMAEkqlAEg\nSYUyACSpUAaAJBXqtGEXoJm1Wi327t3L0aOPDeR8q1efw+LFiwdyLknNYAA01Pj4fq65fidnLD17\n3s/1+KPf4cZ3XMG555437+eS1BwGQIOdsfRsRs9cPuwyJC1QtQIgItYCN9CeM9iemdu6jr8ZeAvQ\nAo4Bv5uZD1bHrgOuAp4ErsnMXf0rX5I0Vz0ngSNiEXATcDnwYmBjRLywq9mHM/OCzLwQuB54b9X3\nRcDrgPOBVwHvi4iRPtYvSZqjOncBXQLsy8wDmXkc2AGs72yQmZ0zlaPAU9XrK4AdmflkZo4D+6r3\nkyQNWZ1LQMuBgx3bE8zwSzwirgbeDjwLeEVH3y91NPtWtU+SNGR1AmCmSzZT3Tsy8320L/FsAN4J\nvLFu325jY0tqlLWwTU6ODvR8y5aNNn7cm17fIDkW0xyLuasTABPAyo7tFcChE7T/KHBzR9/nzaIv\nAEeOHKtR1sI2qPv/O8/X5HEfG1vS6PoGybGY5lhMm0sQ1pkD2A2siYhVEXE6sAHY2dkgItZ0bK4D\n9lavdwIbIuL0iHg+sAa4e9ZVSpL6ruc3gMxsRcRmYBfTt4HuiYitwO7M/BSwOSIuA54AJoFNVd8H\nIuJjwAPAceDqzOx5CUiSNP9qPQeQmXcA0bVvS8frt56g77uBd8+1QEnS/HAxOEkqlAEgSYUyACSp\nUAaAJBXKAJCkQhkAklQoA0CSCmUASFKhDABJKpQBIEmFMgAkqVAGgCQVygCQpEIZAJJUKANAkgpl\nAEhSoQwASSqUASBJhTIAJKlQBoAkFcoAkKRCGQCSVCgDQJIKdVqdRhGxFriBdmBsz8xtXcffBvwO\ncBw4AlyVmQerYy3g34ER4EBmvrp/5UuS5qpnAETEIuAm4FLgELA7Im7PzAc7mt0LXJyZ/xsRvwdc\nD2yojv1PZl7U57olSSepzjeAS4B9mXkAICJ2AOuB7wVAZn6ho/2Xgdd3bI/0oU5JUp/VCYDlwMGO\n7QnaofBM3gR8umP72RFxN/AksC0zb591lZKkvqsTADN9gp+aqWFEvAG4GPjljt0rM/PhiHg+cGdE\nfC0zv3miE46NLalR1sI2OTk60PMtWzba+HFven2D5FhMcyzmrk4ATAArO7ZX0J4L+D4RcRlwHfCy\nzDz+9P7MfLj65zcj4vPAhcAJA+DIkWM1ylrYjh59bODna/K4j40taXR9g+RYTHMsps0lCOsEwG5g\nTUSsAg7Tntzd2NkgIi4EbgYuz8z/6tj/XODxzHwiIs4CXgJ83x1EkqTh6PkcQGa2gM3ALuB+YEdm\n7omIrRGxrmr2HuA5wMcj4r6I+ES1/3zgnoi4D/gs8O6uu4ckSUNS6zmAzLwDiK59Wzpev/IZ+n0J\nuOBkCpQkzQ+fBJakQhkAklQoA0CSCmUASFKhDABJKpQBIEmFMgAkqVAGgCQVygCQpEIZAJJUKANA\nkgplAEhSoQwASSqUASBJhTIAJKlQBoAkFcoAkKRCGQCSVCgDQJIKZQBIUqEMAEkqlAEgSYUyACSp\nUKfVaRQRa4EbaAfG9szc1nX8bcDvAMeBI8BVmXmwOrYJ+FNgCnhXZt7Sv/IlSXPV8xtARCwCbgIu\nB14MbIyIF3Y1uxe4ODN/GvgX4Pqq75nAnwE/C/wcsCUilvavfEnSXNW5BHQJsC8zD2TmcWAHsL6z\nQWZ+ITP/t9r8MrC8en05sCszH83MR4BdwNr+lC5JOhl1AmA5cLBje4LpX/AzeRPw6Wfo+60efSVJ\nA1JnDmBkhn1TMzWMiDcAFwO/PNu+ncbGltQoa2GbnBwd6PmWLRtt/Lg3vb5BciymORZzVycAJoCV\nHdsrgEPdjSLiMuA64GXVpaKn+768q+/nep3wyJFjNcpa2I4efWzg52vyuI+NLWl0fYPkWExzLKbN\nJQjrBMBuYE1ErAIOAxuAjZ0NIuJC4Gbg8sz8r45D/wa8q5r4XQS8Erh21lVKkvqu5xxAZraAzbQn\ncO8HdmTmnojYGhHrqmbvAZ4DfDwi7ouIT1R9J4G/BO4B7gK2VpPBkqQhq/UcQGbeAUTXvi0dr195\ngr4fBD44t/IkSfPFJ4ElqVAGgCQVygCQpEIZAJJUKANAkgplAEhSoQwASSqUASBJhTIAJKlQBoAk\nFcoAkKRCGQCSVCgDQJIKZQBIUqEMAEkqlAEgSYWq9QdhStNqtRgf3z+w861efQ6LFy8e2PkkCQyA\nGY2P7+ea63dyxtKz5/1cjz/6HW58xxWce+55834uSepkADyDM5aezeiZy4ddhiTNG+cAJKlQBoAk\nFcoAkKRCGQCSVKhak8ARsRa4gXZgbM/MbV3HX1odvwC4MjNv6zjWAv4dGAEOZOar+1S7JOkk9AyA\niFgE3ARcChwCdkfE7Zn5YEezA8Am4I9meIv/ycyL+lGsJKl/6nwDuATYl5kHACJiB7Ae+F4AZOZD\n1bGpGfqP9KFOSVKf1QmA5cDBju0J2qFQ17Mj4m7gSWBbZt4+i76SpHlSJwBm+gQ/0yf9Z7IyMx+O\niOcDd0bE1zLzmyfqMDa2ZBZv33+Tk6MDPd+yZaM/8DM3oYamaXp9g+RYTHMs5q5OAEwAKzu2V9Ce\nC6glMx+u/vnNiPg8cCFwwgA4cuRY3befF0ePPjbw83X/zE2ooUnGxpY0ur5BciymORbT5hKEdW4D\n3Q2siYhVEXE6sAHYeYL23/vGEBHPrfoQEWcBLwEemHWVkqS+6xkAmdkCNgO7gPuBHZm5JyK2RsQ6\ngIj4mYg4CLwGuDkivl51Px+4JyLuAz4LvLvr7iFJ0pDUeg4gM+8Aomvflo7X9wDPm6Hfl2g/GyBJ\nahifBJakQhkAklQoA0CSCmUASFKhDABJKpQBIEmF8m8Cq/FarRZ79+4d2NPRq1efw+LFiwdyLmmY\nDAA13vj4fq65fidnLD173s/1+KPf4cZ3XMG555437+eShs0A0CnhjKVnM3rm8mGXIS0ozgFIUqEM\nAEkqlAEgSYUyACSpUAaAJBXKu4CkmlqtFuPj+wd2Pp9H0HwzAKSafB5BC40BIM2CzyNoIXEOQJIK\nZQBIUqEMAEkqlAEgSYUyACSpUAaAJBWq1m2gEbEWuIF2YGzPzG1dx19aHb8AuDIzb+s4tgn4U2AK\neFdm3tKn2iVJJ6HnN4CIWATcBFwOvBjYGBEv7Gp2ANgEfLir75nAnwE/C/wcsCUilvahbknSSapz\nCegSYF9mHsjM48AOYH1ng8x8KDP/g/an/E6XA7sy89HMfATYBaztQ92SpJNUJwCWAwc7tieqfXV0\n9/3WLPpKkuZRnTmAkRn2dX/S72vfsbElNd9+fkxOjg70fMuWjf7Az9yEGpqiKWPRlDpg+P+PNIlj\nMXd1AmACWNmxvQI4VPP9J4CXd/X9XK9OR44cq/n28+Po0ccGfr7un7kJNTRFU8aiKXWMjS1p7L+r\nQXMsps0lCOsEwG5gTUSsAg4DG4CNJ2jf+an/34B3VRO/i4BXAtfOukpJUt/1nAPIzBawmfYE7v3A\njszcExFbI2IdQET8TEQcBF4D3BwRX6/6TgJ/CdwD3AVsrSaDJUlDVus5gMy8A4iufVs6Xt8DPO8Z\n+n4Q+OCcK5QkzQufBJakQvkHYaRTSKvVYu/evQObkPbPUi5sBoB0CvHPUqqfDADpFOOfpVS/OAcg\nSYUyACSpUAaAJBXKAJCkQhkAklQoA0CSCmUASFKhDABJKpQBIEmFMgAkqVAGgCQVygCQpEIZAJJU\nqMatBupa55I0GI0LgN+87iOudS5JA9C4AHCtc0kaDOcAJKlQBoAkFcoAkKRC1ZoDiIi1wA20A2N7\nZm7rOn46cAtwMfBd4MrMfCgiVgF7gAerpl/OzKv7Vbwkae56BkBELAJuAi4FDgG7I+L2zHywo9mb\ngKOZeV5EXAm8B9hQHftGZl7U57olSSepziWgS4B9mXkgM48DO4D1XW3WAx+qXt9KOyyeNnLSVUqS\n+q5OACwHDnZsT1T7ZmyTmS3gkYhYVh1bHRFfiYjPRcQvnWzBkqT+qDMHMNMn+KkebUaqNoeBlZk5\nGREXAZ+IiBdl5mAe9e1h2bJRxsaW/MD+ycnRodfRhBqaoilj0YQ6mlBD0zS9viarEwATwMqO7RW0\n5wI6HQSeBxyKiMXAj2TmZHXsCYDMvDci/hN4AXDvSVXdJ0ePPsaRI8dm3D/sOppQQ1M0ZSyaUEcT\namiSsbElja5vkOYShHUCYDewprqj5zDtyd2NXW0+CWwC7gJeC9wJEBFn0Z4cfioizgHWAPtnXaUk\nqe96zgFU1/Q3A7uA+4EdmbknIrZGxLqq2XbgrIjYB7wVuLba/zLgaxFxH/Ax4M2Z+Ui/fwhJ0uzV\neg4gM+8Aomvflo7X/we8boZ+twG3nWSNktRYrVaL8fHBXdjo5yrGjVsMTpLqaLVajVg+fnx8P9dc\nv/OUXMXYAJB0SmrSL95TdRVjA0DSKetU/cXbFC4GJ0mFMgAkqVAGgCQVygCQpEIZAJJUKO8C0gmd\nyg+5SDoxA0An1KR7rSX1lwGgnrzXWlqYnAOQpEIZAJJUKANAkgplAEhSoQwASSqUASBJhTIAJKlQ\nBoAkFcoHwSTNmkuELAwGgKRZc4mQhcEAkDQnLhFy6qsVABGxFriB9pzB9szc1nX8dOAW4GLgu8CV\nmflQdew64CrgSeCazNzVv/IlSXPVcxI4IhYBNwGXAy8GNkbEC7uavQk4mpnn0Q6K91R9XwS8Djgf\neBXwvogY6V/5kqS5qnMX0CXAvsw8kJnHgR3A+q4264EPVa9vBV5Rvb4C2JGZT2bmOLCvej9J0pDV\nCYDlwMGO7Ylq34xtMrMFPBoRy2bo+60Z+kqShqDOHMBMl2ymarap0/f7PP7od2qUdPJ6nacJdTSh\nhqbU0YQamlJHE2poSh1NqKFJdczWyNTUCX8fExE/D/x5Zq6ttq8FpjongiPi01WbuyJiMXA4M8/u\nbhsRdwBbMvOuvv4UkqRZq/MNYDewJiJWAYeBDcDGrjafBDYBdwGvBe6s9u8EPhwR76V96WcNcHcf\n6pYknaSecwDVNf3NwC7gftqTunsiYmtErKuabQfOioh9wFuBa6u+DwAfAx4A/hW4OjNP/JVDkjQQ\nPS8BSZIWJheDk6RCGQCSVCgDQJIK1ajF4HqtOVSKiFhBe22lHwdawN9n5t8Mt6rhqpYkuQeYyMwr\nhl3PsETEUuAfgJ8EngKuKvW26oh4G+1laJ4Cvg78dmY+MdyqBiMitgPrgG9n5gXVvjOBjwKrgHHg\ndZn56InepzHfAGquOVSKJ4G3Z+aLgF8A3lLwWDztGtp3k5XuRuBfM/N84KeAPUOuZygi4ieA3wcu\nqn4Bnkb7FvVSfID278pO1wKfycygfSv+db3epDEBQL01h4qQmQ9n5ler14/R/p+82CU0qm9Ev0r7\nk2+xImIJ8NLM/ABAtcbWfw+5rGFaDDwnIk4DzgAODbmegcnMLwKTXbs712T7EPDqXu/TpACos+ZQ\ncSJiNfDTtB+yK9V7gXfQYxmRApwDfDciPhAR90bE+yPih4dd1DBk5iHgr4GHaK8x9khmfma4VQ3d\n2Zn5bWh/iATGenVoUgDMet2ghS4iRmmvrnpN9U2gOBHxa7Svc36V9n8jJS8nfhpwEfC3mXkR8DjV\nQ5eliYjn0v7Euwr4CWA0In5juFWdepoUABPAyo7tFRT0la5b9bX2VuCfMvP2YdczRL8IXBER+4F/\nBn4lIm4Zck3DMgEczMx7qu1baQdCiS4D9mfm0Wq1gtuAlwy5pmH7dkT8GEBE/DjQc+W4JgXA99Yc\nqv7C2AbaawmV6h+BBzLzxmEXMkyZ+SeZuTIzz6H938Sdmflbw65rGKqv9wcj4gXVrkspd2L8IeDn\nI+KHqj8ydSnlTYh3fyPeCbyxer0J6PnBsTG3gWZmKyKeXnPo6dtAS/sXCkBE/CLweuDrEXEf7Uth\nf5KZdwy3MjXAH9BeYPFZwH7gt4dcz1Bk5t0RcStwH3C8+uf7h1vV4ETER4CXAz8aEQ8BW4C/Aj4e\nEVfRDsjX9nof1wKSpEI16RKQJGmADABJKpQBIEmFMgAkqVAGgCQVygCQpEIZAJJUKANAkgr1/6On\noYGov4pwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2a68d9e850>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def softmax(n):\n",
    "  return np.exp(n) / np.sum(np.exp(n))\n",
    "\n",
    "s = softmax(n)\n",
    "plt.bar(i, s)\n",
    "np.sum(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining the Train Step\n",
    "\n",
    "Once you have defined the network, you define how to train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the train step to minimize the cross entropy with SGD\n",
    "y_ = tf.placeholder(tf.float32, [None, 10]) # the training labels\n",
    "cross_entropy = -tf.reduce_sum(y_ * tf.log(y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Cross Entropy?\n",
    "\n",
    "Cross Entropy returns **higher value for wrong answers, lower value for correct answers**. Used as a **loss function** for the training to evaluate how much error the network makes.\n",
    "\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "$${\\Huge cross\\_entropy (x, y) = - \\sum   y_{i}  log(x_{i})}$$\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Container object of 10 artists>"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEDCAYAAAA7jc+ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADkdJREFUeJzt3W+Q3PVdwPH3pRErvSs0ZSnThMQpIR8qWhEchlqt1eKE\n1gzxga05WqWAHR80/V/H0nEmk/GBpo4D7VCnQxsZqMVgqdPiDKWZWnWGGUEq4HTS+Eks05AjSI/e\n8Scyo2k4H+ymt24v2d/dbW6Pz71fT7j93fd2P/MlvHfvl90fIzMzM0iS6lo17AEkSaeXoZek4gy9\nJBVn6CWpOEMvScUZekkqbnW/BRGxG9gCPJWZbzjJmk8DbwP+G3hPZj460CklSQvW5BX9bcDmk30z\nIt4GXJCZFwJ/AHx2QLNJkgagb+gz835g+hRLtgJ3dNY+CJwVEa8ZzHiSpMUaxDn6tcDhrttPdI5J\nkpaBQYR+ZI5jXldBkpaJvn8Z28AEcH7X7XXAkX4/NDMzMzMyMtdzhCTNOnDgAL97452ceda5p/2x\nXnj2+3zhT69h06ZNp/2xFmHe4Wwa+pFT3Pk9wPuAuyLiCuCZzHyq7x2OjDA5+XzDh6+t1RpzLzrc\ni1nuRdvU1FHOPOtcRl+1NGeEp6aOLut9b7XG5v0zTd5eeSfwFuDVEfE4sAM4A5jJzFsz896IeHtE\n/Cftt1deN+8pJEmnTd/QZ+Y1DdZsH8w4kqRB85OxklScoZek4gy9JBVn6CWpOEMvScUZekkqztBL\nUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScYZekooz9JJUnKGXpOIMvSQVZ+gl\nqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9JBVn6CWpOEMvScUZekkqztBLUnGGXpKKM/SS\nVJyhl6TiDL0kFWfoJam41U0WRcRVwM20nxh2Z+aunu+fD9wOnN1Zc2Nmfm3As0qSFqDvK/qIWAXc\nAmwGLgbGI+KinmV/DNyVmZcC48BfDnpQSdLCNDl1czlwMDMPZeYxYA+wtWfNi8ArO1+fDTwxuBEl\nSYvR5NTNWuBw1+0J2vHvthPYGxEfAM4ErhzMeJKkxWoS+pE5js303B4HbsvMmyLiCuCvaZ/mOaVW\na6zBw68M7sUs92KWewHT06NL+nhr1oyW2/cmoZ8A1nfdXgcc6VlzA+1z+GTmAxHx8og4JzOfPtUd\nT04+P59Zy2q1xtyLDvdilnvRNjV1dMkfbznv+0KehJqco38I2BgRGyLiDGAbcE/PmkN0TtdExOuB\nn+wXeUnS0ugb+sw8DmwH9gL7gD2ZuT8idkbEls6yjwHvjYhHgS8C156ugSVJ89PoffSZeR8QPcd2\ndH29H/jlwY4mSRoEPxkrScUZekkqztBLUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO\n0EtScYZekooz9JJUnKGXpOIMvSQVZ+glqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9JBVn\n6CWpOEMvScUZekkqztBLUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScaubLIqI\nq4CbaT8x7M7MXXOseSewA3gR+PfMfPcgB5UkLUzfV/QRsQq4BdgMXAyMR8RFPWs2An8EvDEzfw74\n0GmYVZK0AE1O3VwOHMzMQ5l5DNgDbO1Z817gM5n5HEBmPj3YMSVJC9Xk1M1a4HDX7Qna8e+2CSAi\n7qf95LEzM78+kAklSYvS5BX9yBzHZnpurwY2Am8GrgE+HxGvXORskqQBaPKKfgJY33V7HXBkjjX/\nkpkvAt+LiAQuBP7tVHfcao3NY9Ta3ItZ7sUs9wKmp0eX9PHWrBktt+9NQv8QsDEiNgBPAtuA8Z41\nX+kcuyMizqEd+cf63fHk5PPzm7aoVmvMvehwL2a5F21TU0eX/PGW874v5Emo76mbzDwObAf2AvuA\nPZm5PyJ2RsSWzpqvAz+IiH3APwAfy8zpeU8jSRq4Ru+jz8z7gOg5tqPn9keBjw5uNEnSIPjJWEkq\nztBLUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScYZekooz9JJUnKGXpOIMvSQV\nZ+glqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9JBVn6CWpOEMvScUZekkqztBLUnGGXpKK\nM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScYZekooz9JJU3OomiyLiKuBm2k8MuzNz10nW\n/Tbwt8AvZubDA5tSkrRgfV/RR8Qq4BZgM3AxMB4RF82xbhR4P/DAoIeUJC1ck1M3lwMHM/NQZh4D\n9gBb51j3J8Au4H8GOJ8kaZGahH4tcLjr9kTn2I9ExCXAusy8d4CzSZIGoMk5+pE5js2c+CIiRoCb\ngGv7/IwkaQiahH4CWN91ex1wpOv2GO1z9//Uif55wFcj4up+fyHbao3Nc9y63ItZ7sUs9wKmp0eX\n9PHWrBktt+9NQv8QsDEiNgBPAtuA8RPfzMzngHNP3I6IfwQ+kpmP9Lvjycnn5z1wRa3WmHvR4V7M\nci/apqaOLvnjLed9X8iTUN9z9Jl5HNgO7AX2AXsyc39E7IyILXP8yAyeupGkZaPR++gz8z4geo7t\nOMnaXx/AXJKkAfGTsZJUnKGXpOIMvSQVZ+glqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9\nJBVn6CWpOEMvScUZekkqztBLUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScYZe\nkooz9JJUnKGXpOIMvSQVZ+glqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9JBVn6CWpuNVN\nFkXEVcDNtJ8Ydmfmrp7vfxj4feAYMAlcn5mHBzyrJGkB+r6ij4hVwC3AZuBiYDwiLupZ9jBwWWZe\nAnwZ+PNBDypJWpgmr+gvBw5m5iGAiNgDbAX+48SCzPznrvUPAO8a5JCSpIVrco5+LdB9Gmaic+xk\nbgC+tpihJEmD0+QV/cgcx2bmWhgR7wYuA361yYO3WmNNlq0I7sUs92KWewHT06NL+nhr1oyW2/cm\noZ8A1nfdXgcc6V0UEVcCNwJvzsxjTR58cvL5JsvKa7XG3IsO92KWe9E2NXV0yR9vOe/7Qp6EmoT+\nIWBjRGwAngS2AePdCyLiF4DPApsz8wfznkKSdNr0PUefmceB7cBeYB+wJzP3R8TOiNjSWfZJ4BXA\nlyLikYj4ymmbWJI0L43eR5+Z9wHRc2xH19e/MeC5JEkD4idjJak4Qy9JxRl6SSrO0EtScYZekooz\n9JJUnKGXpOIMvSQVZ+glqThDL0nFGXpJKs7QS1Jxhl6SijP0klScoZek4gy9JBVn6CWpOEMvScUZ\nekkqztBLUnGGXpKKM/SSVJyhl6TiDL0kFWfoJak4Qy9JxRl6SSrO0EtScYZekooz9JJUnKGXpOIM\nvSQVZ+glqThDL0nFGXpJKm51k0URcRVwM+0nht2Zuavn+2cAdwCXAU8Dv5OZjw94VknSAvR9RR8R\nq4BbgM3AxcB4RFzUs+wGYCozL6T9hPDJQQ8qSVqYJqduLgcOZuahzDwG7AG29qzZCtze+fpu4K2D\nG1GStBhNQr8WONx1e6JzbM41mXkceCYi1gxkQknSojQ5Rz8yx7GZPmtG5ljz/xw4cICpqaMNHn7x\nLrjgwpN+77vfPTj0GdyLWcthL5ZqH041A7gX3V549vtLMsOpHme57MVCNAn9BLC+6/Y64EjPmsPA\n+cCRiHgZ8MrMnD7VnW7atGmuJ5Al12pdOuwRaLXGhj0C4F7MzjD8fQD3onuGB7+8POZ4qWoS+oeA\njRGxAXgS2AaM96z5e+Ba4EHgHcA3BzmkJGnh+p6j75xz3w7sBfYBezJzf0TsjIgtnWW7gXMi4iDw\nIeDjp2tgSdL8jMzMnPJUuiTpJc5PxkpScYZekooz9JJUXKNr3Qxav2vnrBQRsY72NYLOA44Dn8vM\nTw93quHpXG7jW8BEZl497HmGKSLOAj4P/CzwInB9Zj443KmWXkR8mPYlVl4Evg1cl5n/O9yplk5E\n7Aa2AE9l5hs6x14F3AVsAL4HvDMznz3V/Sz5K/qG185ZKX4IfCQzfwZ4I/C+FbwXAB8EvjPsIZaJ\nTwH3ZubrgZ8H9g95niUXEa8F3g9c2oncatpv715JbqPdym4fB76RmUH7rew39ruTYZy6aXLtnBUh\nM/8rMx/tfH2U9n/MvZeXWBE6v928nfar2BUtIsaAX8nM2wAy84eZ+dyQxxqWlwGviIjVwJn8+Ic1\nS8vM+4HeD592X1vsduC3+t3PMELf5No5K05E/DRwCe0Pna1ENwF/SJ9LZ6wQrwOejojbIuLhiLg1\nIn5q2EMttcw8AvwF8DjwBPBMZn5juFMtC+dm5lPQfrEItPr9wDBC3+TaOStKRIzSvurnBzuv7FeU\niPhN2ucgH6X952NZXB5jiFYDlwKfycxLgRdYgR9CjIizab963QC8FhiNiGuGO9VL0zBC3+TaOStG\n51fSu4EvZOZXhz3PkLwJuDoiHgP+Bvi1iLhjyDMN0wRwODO/1bl9N+3wrzRXAo9l5lTnE/p/B/zS\nkGdaDp6KiNcARMR5QN8rvg0j9D+6dk7n/0y1DbhnCHMsF38FfCczPzXsQYYlMz+Rmesz83W0/zx8\nMzN/b9hzDUvn1/LDEbGpc+itrMy/pH4cuCIiXh4RI7T3YcX9pTQ//lvuPcB7Ol9fC/R9gbjkb6/M\nzOMRceLaOSfeXrkS/+UREW8C3gV8OyIeoX0K6xOZed9wJ9My8AHgixHxE8BjwHVDnmfJZea/RsTd\nwCPAsc4/bx3uVEsrIu4E3gK8OiIeB3YAfwZ8KSKup/1k+I5+9+O1biSpOD8ZK0nFGXpJKs7QS1Jx\nhl6SijP0klScoZek4gy9JBVn6CWpuP8DAeJzGyPmbC8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2a693dabd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# label\n",
    "label = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0])\n",
    "plt.bar(i, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2a68f93350>]"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEDCAYAAADZUdTgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl81PW97/HXbFlIQoAwBEgAWT8QQFYRtSquFUFwX462\nbrX2WE89x57eh+295/Ta23Nvl9PW9nh6WtfiqUutWgVE60LdFdll/bIKhCUsIYEQsuf+MQMTYyAT\nMskvybyfj0cemeWXmY8ff7znN9/5zffrq6+vR0REkoPf6wJERKT9KPRFRJKIQl9EJIko9EVEkohC\nX0QkiSj0RUSSSDDeDc3MDywBCp1zsxrddyvwc6AwetPDzrknElaliIgkRNyhD9wHrAW6n+D+55xz\n32l9SSIi0lbiGt4xs3zgcuCxk2zmS0hFIiLSZuI90v8V8D0g+yTbXG1m5wIbgPudc4Un2VZERDzQ\n7JG+mc0AipxzK4gczTd1RD8XOM05Nx54G5iT0CpFRCQhfM3NvWNm/xe4BagB0oEs4CXn3NdPsL0f\nKHbO9TjZ49bX19f7fBoREhFpoVYFZ7Oh35CZnQ98t4mzd/o65/ZEL18FfM85d3YzD1e/b9/hltbb\nJYXDWagXEepFjHoRo17EhMNZrQr9lpy98wVm9iCw2Dk3H/iOmc0CqoFi4LbWFCUiIm2jRUf6CaYj\n/SgdxcSoFzHqRYx6EdPaI319I1dEJIko9EVEkohCX0QkiSj0RUSSiEJfRCSJKPRFRJKIQl9EJIko\n9EVEkohCX0QkiSj0RUSSiGehP+/9LXg4BYSISFLyLPQfeXkV76zY5dXTi4gkJc9CPzM9xJ8WbqTo\nYLlXJYiIJB3PQv9bV59OVXUdj89fR12dhnlERNqDZ6F/3oQ8zhjZh007S3n90+1elSEiklQ8C32f\nz8fXvmpkZ6Tw8vtb2LG3zKtSRESSRtyhb2Z+M1tmZnObuC/FzJ4zs41m9rGZDYznMTPTQ9x++Uhq\naut5dN5aqmvqWlK7iIi0UEuO9O8D1p7gvjuJLIY+HHgI+Fm8D3r60N6cP74/hfvKmPvh1haUIyIi\nLRVX6JtZPnA58NgJNpkNzIlefgG4qCVFXH/BMHpnp7Hgk21sKixtyZ+KiEgLxHuk/yvge8CJTrPJ\nA3YAOOdqgRIz6xVvEempQb4xswDq4bH5a6moqon3T0VEpAWCzW1gZjOAIufcCjObBjS1KG/j23yc\n+AXiuHA46wuXr9p5iJfe2cS8j7dzz7XjmvvzLqVhL5KdehGjXsSoF4nRbOgD5wCzzOxyIB3IMrOn\nnHNfb7DNDmAAsMvMAkB359zB5h648er2X52cx6I1u3nt488ZNSCbMUNy4v3v6NTC4awv9SJZqRcx\n6kWMehHT2he/Zod3nHM/cM4NdM4NAW4EFjYKfIB5wK3Ry9cBC0+lmFAwwF0zCwj4fTyxYB1lR6tP\n5WFEROQETvk8fTN70MxmRq8+DvQ2s43APwIPnOrjDszNYvZXBlNSVsUf33Cn+jAiItIEn4czXdaf\n6O1abV0dP3l6GZt3HuLuWaM5syC3nUtrX3rrGqNexKgXMepFTDic1dTnqnHrkPPpB/x+vjGjgJSQ\nnz++4Th4uNLrkkREuoQOGfoAub26ccMFwzhSUcOTC9Zp7n0RkQTosKEPMG1CHmMG92L11mLNvS8i\nkgAdOvR9Ph+3Xz6KjLRgZO79Ys29LyLSGh069AF6ZqVyy6VGVXUdj726lto6TcomInKqOnzoA5xZ\nkMuUUX3YvPMQr32iufdFRE5Vpwh9gFsuNXpkpvDKB1vZtkenbomInIpOE/qZ6SHumDGK2rp6Hp2/\nluqaWq9LEhHpdDpN6AOMGZzDBRPz2LX/CC++u8XrckREOp1OFfoA108bRm7PdN5cvIN125qd001E\nRBrodKGfmhLgG1cU4PP5ePzVtZRXaO59EZF4dbrQBxjaP5uZZw+i+FAlz7y1wetyREQ6jU4Z+gAz\nzz6N0/pm8dHqPSxZv9frckREOoVOG/rBgJ+7riggFPTz1F8dJWWalE1EpDmdNvQB+uVkcN20oZQd\nreYPr63XpGwiIs3o1KEPcOGkfEaf1pPPNh/gXU3KJiJyUvEsjJ4KvAekRLd/wTn3YKNtbgV+DhRG\nb3rYOfdEgmttkt/n444ZBfzr44t4buFGRg3qSW6vbu3x1CIinU48a+RWAhc45yYA44HpZjaliU2f\nc85NjP60S+Af0zMrla99NTIp26PzNSmbiMiJxDW845w7NqdxKpGj/aYGz1u1hFdrTRmVy9SCXLbs\nOsSrH2/zshQRkQ4rrtA3M7+ZLQf2AG865xY3sdnVZrbCzJ43s/yEVhmnmy8dQc+sVOZ+8Dlbdx/y\nogQRkQ6tRQujm1l34GXgXufc2ga39wTKnHPVZnY3cL1z7qJmHq5NTrVZuWEf/+v3H5EXzuSh+88n\nLaXZjy1ERDqTVo2qtCj0AczsX4kE/C9PcL8fKHbO9WjmoerbanX7Z9/ayJtLdnDhxDxuudTa5DkS\nKRzOoq160dmoFzHqRYx6ERMOZ7Uq9Jsd3jGz3maWHb2cDlwMrG+0Td8GV2cDa/HQNecPoX/vDBYu\n28mqLQe8LEVEpEOJZ0y/H/A3M1sBLAL+6pxbYGYPmtnM6DbfMbPV0XH/e4Hb2qbc+KSEAtw1s4CA\n38cTC9ZRdrTay3JERDqMFg/vJFCbDe8c8+rHn/Piu1uYZGHuuXIMPp+nJxidkN66xqgXMepFjHoR\n0+bDO53Z9DMHMTw/m6VuHx+t3uN1OSIinuvSoe/3+/jGzALSUgI8/eYG9pcc9bokERFPdenQBwj3\nSOfvLh5BRVUtj81fS12dJmUTkeTV5UMf4JyxfZk0IsyGwlL++ul2r8sREfFMUoS+z+fj65cZ2Zkp\nvPTeFrYX6QMhEUlOSRH6AFndUrjz8lHU1tXzyLy1VFXXel2SiEi7S5rQBxgzJIeLJuWza/8RXnhn\ns9fliIi0u6QKfYDrpg2lX0433lpayOqt+rauiCSXpAv9lFCAb14xmoDfx+Ov6tu6IpJcki70AQb1\nzeKq84ZQWlbFHK2tKyJJJClDH+CyKQMZMaAHSzfs44NVu70uR0SkXSRt6Ee+rTuK9NQgz7y1kb0H\ny5v/IxGRTi5pQx+gd3Y6X7t0BJVVtVpbV0SSQlKHPsDU0X05syCXzTsPMf8jra0rIl1b0oc+wNcu\nHUGv7qnM+/BzNu8s9bocEZE2o9AHuqWFuGtmAfX19Twybw1HK2u8LklEpE00u2q4maUC7wEp0e1f\ncM492GibFOApYBKwH7jBOdepZjazgT2ZPnUQCz7ZxrNvbeSOGaO8LklEJOGaPdJ3zlUCFzjnJgDj\ngelmNqXRZncSWQx9OPAQ8LOEV9oOrjx3MINys/hg1W6WrN/rdTkiIgkX1/COc+7Y+YypRI72G3+b\naTYwJ3r5BeCihFTXzoIBP9+cVUBK0M+c19dTfKjC65JERBIqrtA3M3900fM9wJvOucWNNskDdgA4\n52qBEjPrldBK20m/nAxuvGg4RypqePzVddTp27oi0oU0O6YP4JyrAyaYWXfgZTMrcM6tbbBJ44V6\nfXz53cCXhMNZcRfanq69xHCFpSxas4cP1xRx9QXD2/w5O2ovvKBexKgXMepFYsQV+sc45w6Z2TvA\nZUDD0N8BDAB2mVkA6O6cO9jc43Xk1e1vumgY6z8v5qkF6xjYO4NBfdtuhwuHszp0L9qTehGjXsSo\nFzGtffFrdnjHzHqbWXb0cjpwMbC+0WbzgFujl68DFraqqg6ge7cU7pwRWXTl93PXUKlFV0SkC4hn\nTL8f8DczWwEsAv7qnFtgZg+a2czoNo8Dvc1sI/CPwANtU277GjMkh0smD2BPcTl/enuj1+WIiLSa\nz8Nphes7w9u16ppa/s+cpRTuK+Peq8cycUQ44c+ht64x6kWMehGjXsSEw1mNP0NtEX0jtxmhYIC7\nZxUQCvr5w2vrOXi40uuSREROmUI/DnnhTK6/YBhlR6t5bP5ancYpIp2WQj9OF07MY9zQHNZtO8gb\nn+7wuhwRkVOi0I+Tz+fj9hmjyM5I4cV3N7Ntj8YXRaTzUei3QMPTOH83dw2VVTqNU0Q6F4V+C40Z\nksOlZwygqLicZ97a4HU5IiItotA/BdecP5SBuZm8/9luFms2ThHpRBT6pyAU9HP3rNGkhPzMeW09\nB0o1G6eIdA4K/VPULyeDv7t4BOWVNTwyb40WVReRTkGh3wrnnt6PyRZmY2GpFlUXkU5Bod8KPp+P\nW6ePJKd7KnM/3MqGHSVelyQiclIK/VbKSAtx1xWjAXhk3hqOVFR7XJGIyIkp9BNgxIAezDpnMMWH\nKvnDgvV4OImdiMhJKfQTZObZgxiRn83SDft4d8Uur8sREWmSQj9BAn4/35w1moy0IM++vZGd+8q8\nLklE5EuaXS7RzPKBp4C+QC3wqHPuN422OR94BdgSvekl59yPE1xrh9erexq3Xz6Kh19axe9eWcO/\n3DqZlFDA67JERI6L50i/BrjfOVcAnAV828xGNrHde865idGfpAv8YyaOCHPBxDx27j/Ccws3eV2O\niMgXNBv6zrk9zrkV0ctlwDogr4lNW7WaS1dy44XDyA9n8s7ynSzRNA0i0oG0aEzfzE4DxhNZK7ex\nqWa23MxeNbOCRBTXWYWCAb41OzJNw5OvrWd/yVGvSxIRAVoQ+maWCbwA3Bc94m9oKTDIOTcBeBh4\nOXEldk79e2dw8yUjOFpZw+/nraGmVtM0iIj34loY3cyCwHzgNefcr+PYfiswyTlXfJLNuvzJ7PX1\n9fzi6WW8u7yQay8czq0zkvoNkIgkRquG0ps9eyfqCWDtiQLfzHKdc0XRy1MAXzOBD5AUq9tfP20I\n67Ye4IWFGxkY7saYwTlf2iYczkqKXsRDvYhRL2LUi5hwOKtVfx/PKZvnADcDq8xsOZEj9B8Ag4B6\n59wjwLVm9vdANXAUuKFVVXUh6alBvnXlaP7tqaU8Nm8t//uOKfTITPW6LBFJUnEN77SR+mR65X5z\n8Q6efXsjowb15Ls3jMfvj71D01FMjHoRo17EqBcx4XBWq4Z39I3cdnLx5HzGD+vNum0Hmf/x516X\nIyJJSqHfTnw+H3fMGEWv7qm88sFW3PaDXpckIklIod+OMtND3D1rND58/H7uGg4dqfK6JBFJMgr9\ndjY8vwdXnz+EkrIqHp2/ljpNwywi7Uih74HLzhzI2CE5rNlazGufaJlFEWk/Cn0P+H0+7pw5ip5Z\nqbz03hbWbDngdUkikiQU+h7p3i3l+Pj+z/57CYfKNb4vIm1Poe+hEQN6cNV5gyk+VMFj8zS+LyJt\nT6HvselTBzF5VC6rtxbz6sca3xeRtqXQ95jf5+OfbppIr+6pvPz+FtZt0/n7ItJ2FPodQPeMFL41\newx+n49H5q6htKzS65JEpItS6HcQw/KyuXbaUEqPVPH7uWuordP8+yKSeAr9DuTSMwYwcUSY9dtL\nePn9rV6XIyJdkEK/A/H5fNxx+Uj69Ejn1Y+3sXLTfq9LEpEuRqHfwXRLC3HPVWMIBvw8Nn+t1tcV\nkYRS6HdAA3OzuOXSERypqOG3L6+mukbj+yKSGPGsnJUPPAX0BWqBR51zv2liu98A04EjwG3OuRUJ\nrjWpnHt6PzYWlvDhqj08+9YGvn7ZSK9LEpEuIJ4j/RrgfudcAXAW8G0z+0ICmdl0YKhzbjhwN/C7\nhFeaZHw+H7dcagzok8k7K3bx4ardXpckIl1As6HvnNtz7KjdOVcGrAPyGm02m8i7AZxzi4BsM8tN\ncK1JJzUU4NtXjSE9NchTf3VsL9JycSLSOi0a0zez04DxwKJGd+UBOxpc38mXXxjkFPTp2Y1vzBxF\ndU0dv/3Lasorqr0uSUQ6sbhD38wygReA+6JH/A01tVCvZg9LkAnDw8w4axB7S47y2Px1mphNRE5Z\nsx/kAphZkEjg/7dz7pUmNikEBjS4ng/sau5xw+GseJ4+KTTXi7uuHsfO/eWs2LiPd1bu5oZLrJ0q\na3/aL2LUixj1IjHiCn3gCWCtc+7XJ7h/LvBt4E9mNhUocc4VNfeg+/ZpjBoiO3M8vbh9uvGjokM8\n/fp6wt1TGTskpx2qa1/x9iIZqBcx6kVMa1/8mh3eMbNzgJuBC81suZktM7PLzOxuM/smgHNuAbDV\nzDYBvwfuaVVV0qSsbincc9VYAgE/j8xdwz59cUtEWshX7934cL1euSNaehTz3spd/OG19Qzok8kP\nvjaJ1FCgDatrXzqii1EvYtSLmHA4q6nPUOOmb+R2QueN68+08f3ZsbeMOa+vx8MXbhHpZBT6ndRN\nF49gaF53PllTxJtLCr0uR0Q6CYV+JxUK+rnnyrFkZ6Tw/MJNWnFLROKi0O/Eemalcs9VY/D54L9e\nXs3+Un2wKyInp9Dv5Ibn9+DmS0ZQdrSah19cRWVVrdcliUgHptDvAqZNyGPa+P5s31vGk6+t0we7\nInJCCv0u4u8uGcGw/Gw+XbeXBZ9s87ocEemgFPpdRDDg59tXjqFnViovvbtFSy2KSJMU+l1IdmYq\n9149lmDQz+/nrmHn/iNelyQiHYxCv4sZ3K87t18+koqqWv7jhc8oO6qpmEUkRqHfBU0t6Ht8Kub/\nenk1NbVaY1dEIhT6XdRV5w1h/LDerNt2kGff3uh1OSLSQSj0uyi/z8ddVxSQH87gb8t28vZSTdUg\nIgr9Li09Nch3rj2d7t1CPPvWRlZvPeB1SSLiMYV+F9c7O517rzkdv9/Hf728hl06o0ckqSn0k8Cw\nvGxuv3wkRytr+PULKzlUXuV1SSLikWaXSzSzx4GZQJFz7vQm7j8feAXYEr3pJefcjxNapbTaWaP7\nsudAOfM++pyHX1zF924aTyjYdRZfEZH4xLNG7pPAfwBPnWSb95xzsxJTkrSVK88dTNHBcj5dt5cn\nFqznm1cU4PO1ahEeEelkmh3ecc59ADQ3WbuSoxPw+XzcOWMUw/KyWbS2iFc+2Op1SSLSzhI1pj81\numj6q2ZWkKDHlDYQCga495qxhHukMffDz/ngs91elyQi7Sie4Z3mLAUGOefKzWw68DIwIp4/DIez\nEvD0XUN79iIMPPjNs/kf//E+c15fz+ABPRg/ok+7PX9ztF/EqBcx6kVi+OKZe93MBgHzmvogt4lt\ntwKTnHPFzWxar9XtI8LhLLzoxYYdJfz7c8sJBvx8/5ZJDOiT2e41NOZVLzoi9SJGvYgJh7NaNZwe\n7/COjxOM25tZboPLUwBfHIEvHcCIAT34xswCKqpqeejPKyk+VOF1SSLSxuI5ZfMZYBqQY2bbgR8C\nKUC9c+4R4Foz+3ugGjgK3NB25UqiTRmVy4FDFfz5b5v51Z9X8v2bJ9ItLeR1WSLSRuIa3mkjGt6J\n8vqta319Pc+8uZG3lxUyYkAPvnvDOM/O4fe6Fx2JehGjXsS01/COdGE+n4+bLh7OZAuzYUcJj85b\nS12d1tkV6YoU+gKA3x+ZlXPEgB4scft49q2NWmBdpAtS6MtxoWCA71wzlrxwBm8vK2TeR597XZKI\nJJhCX76gW1qI+68fT073NF5+fyt/W6Z5+EW6EoW+fEnPrFT++cbxZHUL8cc3NvDpuiKvSxKRBFHo\nS5Nye3Xj/uvHk5oS4NF5a1m1RQuwiHQFCn05oUF9s7jv2sgCLP/50io27CjxuiQRaSWFvpyUDezJ\nPVeOobaunof+vJKtuw95XZKItIJCX5o1blhv7rqigMrqWn71/Ep27ivzuiQROUUKfYnLlFG53HbZ\nSMqOVvPz51aw+4DW2hXpjBT6Erdzx/Xn5ktGcOhIFT9/djl7D5Z7XZKItJBCX1rkokn5XH/BMErK\nIsG/v/So1yWJSAso9KXFLjtzIFefN4QDhyr52TPLOVCqKZlFOguFvpySmWefxuyvDGZ/aQU/fWaZ\njvhFOgmFvpyy2V8ZfDz4f/aMhnpEOgOFvrTK7K8M5soGwb+vRMEv0pE1G/pm9riZFZnZZyfZ5jdm\nttHMVpjZ+MSWKB3drK8M5qpzI8H/k6eXUVSss3pEOqp4jvSfBL56ojvNbDow1Dk3HLgb+F2CapNO\n5IpzBnPdBUM5eLiSnzy9TF/gEumgmg1959wHwMGTbDIbeCq67SIgu+Fi6ZI8pp85iJsvGUHpkSp+\n+sxytu3R8nYiHU0ixvTzgB0Nru+M3iZJ6KJJ+dw2fSRHjlbzs2eXaZI2kQ4mmIDHaGqR3rjW2QuH\nsxLw9F1DV+rFNRcb4ZwMfvnMMn75pxU8cOsZnFHQN+6/70q9aC31Ika9SIxEhH4hMKDB9XxgVzx/\nqNXtI8LhrC7Xi1H52fzDNafz27+s4t+e/JQ7ZozirNHNB39X7MWpUi9i1IuY1r74xTu846PpI3qA\nucDXAcxsKlDinNNSS8LpQ3P47o3jSQlFFmL566fbvS5JJOk1e6RvZs8A04AcM9sO/BBIAeqdc484\n5xaY2eVmtgk4AtzelgVL5zI8vwcP3DyRXz2/gj8t3MTBw5Vcf+Ew/L4THUOISFvy1dfHNfzeFur1\ndi0iGd66Hiit4JfPr2D3gXKmjOrDnTMKCAW//EYzGXoRL/UiRr2ICYezWnXEpG/kSrvIyU7j+7dM\nYlh+Np+u28svnltO2dFqr8sSSToKfWk3mekh/vmG8Zwxsg8bCkv5t6eW6Nu7Iu1MoS/tKiUU4O7Z\no5lx1iCKDh7lx08twW0/2Xf/RCSRFPrS7vw+H9ecP5Tbp4+koqqWf39uBe+u2Ol1WSJJQaEvnjl3\nXH++e8N40lODzHnd8fQbG6iprfO6LJEuTaEvnho5qCf/cutk8sIZvL2skB8+8jGHyqu8Lkuky1Lo\ni+fCPdL5wS2TmDC8N59t2s+P/rCYrbsPeV2WSJek0JcOIT01yLevHsst00dy8FAl/++PSzXOL9IG\nFPrSYfh9Pm642Pin68eRGgow53XHY/PXUllV63VpIl2GQl86nDFDcvjhbWcwuF8WH63ew4/mLKZQ\ni7KIJIRCXzqk3j3S+f4tk7hk8gB2Hyjnx3OW8N7KXXg4bYhIl6DQlw4rGPBz08XDuffqsQQDfv7w\n2np++5fVmr5BpBUSMZ++SJuaOCLMoNwsHpu/lqUb9rF5Vyl3zixg9Gm9vC5NpNPRkb50CjnZaXzv\npglcc/4QDpdX84vnVvDHN5w+5BVpIYW+dBp+v48ZZ53G//z6JPr3zmDhsp386xOLtA6vSAso9KXT\nOa1vd35422SmTx3I/tIKfvr0Mv74huNoZY3XpYl0eHGN6ZvZZcBDRF4kHnfO/bTR/bcCPyeyXi7A\nw865JxJZqEhDoWCA66YNY8LwME8uWMfCZTtZvnE/X7vUGD+8t9fliXRYza6cZWZ+YANwEZEFzxcD\nNzrn1jfY5lZgknPuOy14bq2cFaVVgWJOpRfVNXW8+vHnvPrxNmrr6plkYW68cDg52WltU2Q70X4R\no17EtHblrHiO9KcAG51z2wDM7DlgNrC+0XZa9FQ8EQr6ufLcIZwxsg9z/upY6vaxassBrjj7NL46\nZSDBgEYxRY6J519DHrCjwfXC6G2NXW1mK8zseTPLT0h1Ii2QF87kgZsncueMUaSGArz47hb+5fFP\nWbFpv77UJRIVz5F+U0fwjf8FzQWecc5Vm9ndwBwiw0EnFQ5nxfH0yUG9iGltL668sDsXnzWYp19b\nx4KPtvKbFz5j/PAwd8wazeD+2Qmqsn1ov4hRLxIjntAvBAY2uJ5PZGz/OOdcw/XuHgW+8EHviWiM\nLkLjlTGJ7MXV5w7mzJFh/rRwEys27uO+X77DOWP6MfsrgzvFeL/2ixj1Iqa1L37xhP5iYJiZDQJ2\nAzcCNzXcwMz6Ouf2RK/OBta2qiqRBMkLZ3L/DeNZteUAz/9tEx+s2s0na/dwwYR8Zpw9iO7dUrwu\nUaRdNRv6zrlaM7sXeIPYKZvrzOxBYLFzbj7wHTObBVQDxcBtbVizSIuNHZLD6NN68cnaPfzlva28\nuWQH763cxYUT8/jqmQMV/pI0mj1lsw3plM0ovXWNaY9eVNfU8e6Knbz6yTZKy6pICfm5cEI+l5wx\ngJ5ZqW363C2h/SJGvYhpj1M2RbqUUNDPxZMHcP74/ry3cjevfvw5r3+6nbeW7uDsMX257MxB9O3V\nzesyRdqEQl+SVigY4KJJ+Zw3rh8frt7D64u2897K3by/cjfjhvXmksn5jBzUE59PX0GRrkOhL0kv\nFAwwbXwe553en2Ub9vHaou2s2LSfFZv2kx/O4MJJ+UwtyCUtRf9cpPPTmH4HoPHKmI7Si807S3lz\nyQ6Wun3U1tWTlhJg6ui+TBvfn4G57XO+eEfpRUegXsRoTF+kDQzNy2ZoXjYHD1fy/spdvLtyF+8s\n38k7y3cyMDeTc8b2Y2pBLlk660c6GR3pdwA6ionpqL2oravjs80HeH/lbj7bfIC6+noCfh9jh+Qw\npaAPE4aFSU0JJPQ5O2ovvKBexOhIX6QdBPx+JgwPM2F4mNIjVSxas4cPV+85PvafEvIzbmhvJlmY\n04fmaPxfOiztmSItlJ2RwqVTBnLplIHs2n+ERWuLWLSuiMXr97J4/V6CAT9jBvdi/PDejB2S06HO\n/RfR8E4HoLeuMZ21F/X19ezYW8ayDftY6vaxc/+R4/cN6pvF2CG9GDM4hyH9u8c91XNn7UVbUC9i\nNLwj0gH4fD4G5mYxMDeLK88dQtHBcj7bdICVm/fjtpewbc9h5n+0jbSUACMH9mTkwB7YwJ4MyM3E\nr+8BSDtS6Iu0gdye3bjkjG5ccsYAjlbW4LaXsGZrMau3Hjj+OQBARlqQoXnZDM/PZnh+D07rm0VK\nKLEfCIs0pNAXaWPpqUHGD+99fO3e4kMVrN9+kPXbSli//SCfbT7AZ5sPABDw+8gLZzCkfzbjRoTp\n2S1E/94ZWv1LEkZj+h2AxitjkrEXJWWVbCosZWNhKVt2l7JtTxk1tXXH7w8GfOSFMxnQJ/oTziQv\nnJFU3xFIxv3iRDSmL9LJ9chMZfLIPkwe2QeAmto6duwt40BZFas37Wdb0WF27itj254vhl5WtxD9\nczLol9ONNz6FAAAHg0lEQVSNvr26kdsr8jsnO03vDOSEFPoiHUww4Gdwv+5MCWcxOTokVFNbR1Fx\nOTv2lVG49wi79h9h14EjbNhRgttR8oW/9/t85GSn0qdHOuEe6eRkp9E7O/K7V1YqPTJT8fv14XGy\niiv0zewy4CFii6j8tNH9KcBTwCRgP3CDc257gmsVSVrBgJ+8cCZ54UwoiN1eVV1L0cGjFBWXs7u4\nnKLicvaWHGXvwaOs+fwgcPBLj+X3+eiRlULPzFR6ZKUe/52dkUJ2Rgrdoz+Z6SG9Y+iCmg19M/MD\nDxNZ6HwXsNjMXnHOrW+w2Z1AsXNuuJndAPyMyLKKItKGUkKB42P9jVVU1XCgtIL90Z8DpRUUH66g\n+FAlxYcr+HzPYWp3HTrp42ekBcnqlkJmtxBZ6SEy0kNkpoXISA+SkRaiW1rsd7fUIOnRn1BQLxYd\nVTxH+lOAjc65bQBm9hyRdXAbhv5s4IfRyy8QeZEQEQ+lpQRj7w6aUFdfz+HyakoOV1J6pJKSsipK\nj1RxqKyKQ+VVHC6v4lB5NWXlVRQdLKcl53wEAz7SUoKkpwZISwmSlhIgNSVyOTXkJy0UJCXFT2oo\nQEowQGrIT0ooQEooQCjoJyXoJyUYuRwK+qn1+zlcVkkwELkeCvg1RHWK4gn9PGBHg+uFRF4Imtwm\nuqZuiZn1cs4VJ6ZMEUk0v893fEgHTj5ddF19PeUVNRw5Wk1ZRTVHjtZwpKKa8ooayiuqOVJRw9HK\nGsorI7+PVtZSUVVDRVUt+0qOUllVS6LPE/T7fASDPkIBP4GAn1DARyDgJ+D3EYz+DgR8BPx+ggEf\nfr+PoD92u9/vI+CL/vZHfvv9Pvy+2HWfz4ffR+x+X/Q2f+T2Y5d9PqL3xX5H/vbLt/mI/MbH8ceI\nXI3cduz+yG8IBv0M7ts9YS9y8YR+U8/U+P9f4218TWwjIp2U3+cjMz1EZnqI3FP4+/r6eqpq6qio\nqqWyupaqqloqqmupqo5er66jsrqW6po6qmoi16troj+1dfgDfsqOVB6/XlNTR01tPdW1ddTWRi7X\n1NZRVV1LbV09NbX11NbWUVtXT21d54+imy8ZwUWT8hPyWPGEfiEwsMH1fCJj+w3tAAYAu8wsAHR3\nzn35E6Qv8oXD7bMYRWegXsSoFzHqhSRaPKG/GBhmZoOA3UQ+oL2p0TbzgFuBRcB1wMJEFikiIonR\n7Efszrla4F7gDWAN8Jxzbp2ZPWhmM6ObPQ70NrONwD8CD7RVwSIicuq8nIZBRETamU6mFRFJIgp9\nEZEkotAXEUkinky41txcPl2ZmeUTmaeoL1ALPOqc+42Z9QT+BAwCPgeud86VelZoO4lO87EEKHTO\nzTKz04DngJ7AMuBrzrkaD0tsN2aWDTwGjAHqgDuADSTZfmFm/0Rkapc6YBVwO9CfJNkvzOxxYCZQ\n5Jw7PXrbCfPBzH4DTAeOALc551ac7PHb/Ui/wVw+XwVGAzeZ2cj2rsNDNcD9zrkC4Czg29H//geA\nt5xzRuSU1+97WGN7ug9Y2+D6T4FfRPtQQuQff7L4NbDAOTcKGEdkqpOk2i/MrD/wD8DEaOAFiZwi\nnkz7xZNE8rGhJvcDM5sODHXODQfuBn7X3IN7MbxzfC4f51w1kVfv2R7U4Qnn3J5jr8TOuTJgHZEv\nvM0G5kQ3mwNc6U2F7Sf6rudyIke3x1wIvBi9PAe4qr3r8oKZZQHnOueeBHDO1USP5JJuvwACQIaZ\nBYF0Il8GvYAk2S+ccx/w5elRG+8Hsxvc/lT07xYB2WZ20i9NexH6Tc3lk+dBHZ6LDmWMBz4Bcp1z\nRRB5YQDCHpbWXn4FfI/olB1mlgMcdM4dWzaqkMjb+mQwBNhvZk+a2TIze8TMupFk+4VzbhfwC2A7\nsBMoJTKcU5Kk+8UxfRrtB32itzfO0500k6dehH48c/l0eWaWSWRG0vuiR/xJ1QMzm0FkzHIFsX3C\nx5f3j2TpSxCYCPync24ikfHZB0ie/34AzKwHkaPXQUSCPYPIeHVjSdWXk2hxnnoR+vHM5dOlRd+2\nvgD8t3PulejNRcfelplZX2CvV/W1k3OAWWa2BXiWyLDOQ0Tenh7bL5Np3ygEdjjnlkSvv0jkRSDZ\n9ouLgS3OueLobAB/Ac4GeiTpfnHMifaDQiLznh3TbG+8CP3jc/lEV9y6EZjrQR1eegJY65z7dYPb\n5gK3RS/fCrzS+I+6EufcD5xzA51zQ4jsAwudc7cAfyMyfxMkQR+Oib5132FmI6I3XURk2pOk2i+I\nDOtMNbM0M/MR60Oy7ReN3/U23A9uI/bfPxf4OoCZTSUyDFZ00gf2YhqG6CmbvyZ2yuZP2r0Ij5jZ\nOcB7RE5Fq4/+/AD4FHieyKv2duA651zJiR6nKzGz84HvRk/ZHEzs1LzlwC3RD/y7PDMbR+RD7RCw\nhcipigGSbL8wsx8SORCoJrIPfIPIEWxS7Bdm9gwwDcgBiogsUPUy8Gea2A/M7GHgMiJDgrc755ad\n7PE1946ISBLRN3JFRJKIQl9EJIko9EVEkohCX0QkiSj0RUSSiEJfRCSJKPRFRJKIQl9EJIn8f6Qz\nb+7WJsnvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2a690d4fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def cross_entropy(values, labels):\n",
    "  return -np.sum(labels * np.log(values))\n",
    "\n",
    "# simulate the training\n",
    "x_ents = []\n",
    "for i in range(0, 100):\n",
    "  x_ents.append(cross_entropy(softmax(n), label))\n",
    "  n[8] += 0.1\n",
    "plt.plot(x_ents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Gradient Decent to find the optimal weights\n",
    "TensorFlow finds the \"gradient\" with differentiation and move weights and biases towards the dirrection with less error.\n",
    "![](https://dl.dropboxusercontent.com/u/4207279/public/public_images/Screen%20Shot%202017-02-18%20at%2011.13.57%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training with mini batch\n",
    "\n",
    "Randomly selects 100 samples from the 55K images and calculates the gradient. Repeats 1K times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initialize variables and session\n",
    "init = tf.global_variables()\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "\n",
    "# train the model mini batch with 100 elements, for 1K times\n",
    "for i in range(1000):\n",
    "  batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "  sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test\n",
    "\n",
    "Let's see how accurate the model is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9015\n"
     ]
    }
   ],
   "source": [
    "# evaluate the accuracy of the model\n",
    "correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Today We Learned\n",
    "\n",
    "- **Single neuron** can take an image and recognize a digit\n",
    "- **Matrix operation** (y = Wx + b) can define a neural network concisely\n",
    "- **Softmax** converts the result to probabilities (0.0 - 1.0)\n",
    "- **Cross Entropy** evaluates how much error the network makes (the loss function)\n",
    "- **Gradient Decent** moves weights and biases toward the direction with less error\n",
    "- **Mini Batch** trains the network with small batch of data"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}