SOL1_nnModule.ipynb

{
 "metadata": {
  "language": "lua",
  "name": "",
  "signature": "sha256:cb25ca74e5a3e6cacd11c35857459a96bb7f6f64d8c9bbba8712e00e90503208"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Tutorial by Tom Sercu"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I learned torch about a year ago. I am not as advanced as Soumith by far.\n",
      "But for me the learning process is fresh in memory, so I want to hold your hand to take you through some critical and insightful part of my learning process as I remember it."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Parallels (roughly)\n",
      "+ python -> lua\n",
      "+ numpy -> torch\n",
      "+ sklearn -> nn"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The docs: https://github.com/torch/nn  You HAVE to know this URL by heart. Best documentation is often the code."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "require 'nn' -- auto imports torch too. \n",
      "-- quick hint, You can do \"require 'cunn'\" if on a gpu machine, then you have all you need."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Exercise 1: nn.Linear"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Below I make a single linear layer. What is the size of one input sample? What is the batch size? What is the output size? Print the weights and the bias."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net = nn.Linear(3,4)\n",
      "net:zeroGradParameters() --i'll come back to this later\n",
      "inp = torch.randn(2,3)\n",
      "out = net:forward(inp)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(out:size())\n",
      "print(net.weight)\n",
      "print(net.bias)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "\n",
        " 2\n",
        " 4\n",
        "[torch.LongStorage of size 2]\n",
        "\n",
        " 0.2552  0.0386  0.0651\n",
        " 0.4113 -0.0637  0.0510\n",
        "-0.1045  0.4843  0.1841\n",
        "-0.3314 -0.0864 -0.2937\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n",
        "-0.1491\n",
        " 0.5034\n",
        " 0.2731\n",
        " 0.3178\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ok so now imagine calculating a loss and receiving a gradient to the output of the \"net\" layer. This gradient needs to have the same size as the output: the derivative of a scalar (=the objective) with respect to each of the net's outputs.\n",
      "\n",
      "Backpropagate through the network."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gradientout = torch.randn(2,4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gradientinp = net:backward(inp, gradientout)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What is the size of the gradient wrt the network's inputs?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gradientinp:size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "\n",
        " 2\n",
        " 3\n",
        "[torch.LongStorage of size 2]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "During backpropping through the linear layer, not only the gradient wrt input is calculated, also the gradient wrt the parameters and the bias. Find these gradients."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(net.gradBias)\n",
      "print(net.gradWeight)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "-1.0028\n",
        "-3.6087\n",
        "-1.3583\n",
        "-0.7636\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n",
        "-2.4309 -3.6554 -0.6389\n",
        " 1.8470  6.2386 -3.7559\n",
        "-0.0670  0.9531 -1.3089\n",
        " 0.5779  1.6625 -0.8205\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Take a look at the net's weight and the bias. Scroll up and take a look at the weight & bias before backprop. Explain"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(net.weight)\n",
      "print(net.bias)\n",
      "-- unchanged! \n",
      "-- Note that backprop calls \"updateGradInput\" (replaces gradInput)\n",
      "-- And \"accGradParameters\" --> Just accumulates! \n",
      "-- YOU are responsible for using these gradients and doing something with them."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        " 0.2552  0.0386  0.0651\n",
        " 0.4113 -0.0637  0.0510\n",
        "-0.1045  0.4843  0.1841\n",
        "-0.3314 -0.0864 -0.2937\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n",
        "-0.1491\n",
        " 0.5034\n",
        " 0.2731\n",
        " 0.3178\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Do the exact same backprop step and look at the gradBias and gradWeight again. Explain"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gradientinp = net:backward(inp, gradientout)\n",
      "print(net.gradBias)\n",
      "print(net.gradWeight)\n",
      "-- the gradients have doubled. I wasn't joking about accumulate."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "-2.0057\n",
        "-7.2175\n",
        "-2.7166\n",
        "-1.5272\n",
        "[torch.DoubleTensor of dimension 4]\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "\n",
        " -4.8618  -7.3108  -1.2778\n",
        "  3.6939  12.4772  -7.5119\n",
        " -0.1339   1.9062  -2.6178\n",
        "  1.1558   3.3251  -1.6409\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To make sure you understand what's going on with an update of the weights, let's first set all weights to zero. Now apply an update with learning rate 0.1 and print the new weights."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net.weight:zero()\n",
      "net.bias:zero()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net.weight:zero()\n",
      "net.bias:zero()\n",
      "net:updateParameters(0.1)\n",
      "print(net.weight)\n",
      "print(net.bias)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        " 0.4862  0.7311  0.1278\n",
        "-0.3694 -1.2477  0.7512\n",
        " 0.0134 -0.1906  0.2618\n",
        "-0.1156 -0.3325  0.1641\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n",
        " 0.2006\n",
        " 0.7217\n",
        " 0.2717\n",
        " 0.1527\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net.weight:zero()\n",
      "net.bias:zero()\n",
      "-- net.weight = net.weight - net.gradWeight * 0.1\n",
      "-- net.bias   = net.bias - net.gradBias * 0.1\n",
      "-- Better to use an in-place operation:\n",
      "net.weight:add(-0.1, net.gradWeight)\n",
      "net.bias:add(-0.1, net.gradBias)\n",
      "print(net.weight)\n",
      "print(net.bias)\n",
      "-- idem as above but I did it myself."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        " 0.4862  0.7311  0.1278\n",
        "-0.3694 -1.2477  0.7512\n",
        " 0.0134 -0.1906  0.2618\n",
        "-0.1156 -0.3325  0.1641\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n",
        " 0.2006\n",
        " 0.7217\n",
        " 0.2717\n",
        " 0.1527\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "params, grads = net:getParameters()\n",
      "params:zero()\n",
      "--params = params - grads * 0.1\n",
      "--   THIS DOESNT WORK because you replace \n",
      "--   params with a pointer to a new tensor, i.e. the operation is not in place.\n",
      "params:add(-0.1, grads) -- this is in-place.\n",
      "print(net.weight)\n",
      "print(net.bias)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        " 0.4862  0.7311  0.1278\n",
        "-0.3694 -1.2477  0.7512\n",
        " 0.0134 -0.1906  0.2618\n",
        "-0.1156 -0.3325  0.1641\n",
        "[torch.DoubleTensor of dimension 4x3]\n",
        "\n",
        " 0.2006\n",
        " 0.7217\n",
        " 0.2717\n",
        " 0.1527\n",
        "[torch.DoubleTensor of dimension 4]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ok, now we did our gradient udpate and want to process a new minibatch. Clear the gradient of the bias and gradient of the weights."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net:zeroGradParameters()\n",
      "-- equivalent: net.gradWeight:zero(); net.gradBias:zero()\n",
      "print(net.gradWeight)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "(1,1,.,.) = \n",
        " 0 0 0\n",
        " 0 0 0\n",
        " 0 0 0\n",
        "\n",
        "(1,2,.,.) = \n",
        " 0 0 0\n",
        " 0 0 0\n",
        " 0 0 0\n",
        "[torch.DoubleTensor of dimension 1x2x3x3]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Convolutional layer"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "At home: do all the previous exercises for a convolutional layer.\n",
      "\n",
      "nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH])\n",
      "\n",
      "I make batchsize 1 and output feature maps 1 so you can still print stuff easily."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net = nn.SpatialConvolution(2,1,3,3)\n",
      "inp = torch.randn(1,2,5,5) -- batch of 1, 2 input channels, 5x5 image\n",
      "out = net:forward(inp)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}