3_fullNetwork.ipynb

{
 "metadata": {
  "language": "lua",
  "name": "",
  "signature": "sha256:9cdc3f6361e2a278b1681e2b3064fdee795fb71a760db2cf3ba4aebca482d65d"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You have explored simple layers and the two most popular criterions. Now let's put things together and do one forward & backward pass through a network."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "require 'nn'; torch.manualSeed(1238)\n",
      "input = torch.randn(16, 50) --batchsize, inputsize\n",
      "labels = torch.rand(16):mul(3):add(1):floor() -- 3 class labels"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Make a network with a linear layer (50->h), a ReLU, another linear layer with output size 3, and a logsoftmax."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model = nn.Sequential()\n",
      "-- model:add(...)\n",
      "print(model:__tostring())\n",
      "crit = nn.ClassNLLCriterion()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Do a forward and backward pass. Set the gradients to zero first (they get initialized on a chunk of memory so will contain garbage). Forward through the network. Check that the output looks like log probabilities. Calculate & print the NLL loss. Calculate the gradient wrt the log probabilities. Backprop this gradient through the network. Done"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Look at the gradWeight and gradInput of the second linear layer. Now zero the gradWeight & gradBias and calculate the gradient again by backpropping the gradInput of the layer above (using as input: the output of the layer below). This is what sequential does: https://github.com/torch/nn/blob/master/Sequential.lua"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "See also soumith's tutorial from nextml:\n",
      "\n",
      "http://nbviewer.ipython.org/github/soumith/nextml/blob/master/04-neural-networks-basics.ipynb"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}