chapter02-quickstart/chapter2: PyTorch快速入门.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 PyTorch第一步\n",
    "\n",
    "PyTorch的简洁设计使得它入门很简单，在深入介绍PyTorch之前，本节将先介绍一些PyTorch的基础知识，使得读者能够对PyTorch有一个大致的了解，并能够用PyTorch搭建一个简单的神经网络。部分内容读者可能暂时不太理解，可先不予以深究，本书的第3章和第4章将会对此进行深入讲解。\n",
    "\n",
    "本节内容参考了PyTorch官方教程[^1]并做了相应的增删修改，使得内容更贴合新版本的PyTorch接口，同时也更适合新手快速入门。另外本书需要读者先掌握基础的Numpy使用，其他相关知识推荐读者参考CS231n的教程[^2]。\n",
    "\n",
    "[^1]: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n",
    "[^2]: http://cs231n.github.io/python-numpy-tutorial/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor\n",
    "\n",
    "Tensor是PyTorch中重要的数据结构，可认为是一个高维数组。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）以及更高维的数组。Tensor和Numpy的ndarrays类似，但Tensor可以使用GPU进行加速。Tensor的使用和Numpy及Matlab的接口十分相似，下面通过几个例子来看看Tensor的基本使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.1.0a0+65d6f10_2_ged1fa68'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " from __future__ import print_function\n",
    "import torch as t\n",
    "t.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [3., 4.]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 构建 5x3 矩阵，只是分配了空间，未初始化\n",
    "x = t.Tensor(5, 3)\n",
    "\n",
    "x = t.Tensor([[1,2],[3,4]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.9971, 0.4956, 0.7326],\n",
       "        [0.5040, 0.2773, 0.1133],\n",
       "        [0.1560, 0.8798, 0.3876],\n",
       "        [0.8097, 0.8484, 0.6488],\n",
       "        [0.5509, 0.5815, 0.1071]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用[0,1]均匀分布随机初始化二维数组\n",
    "x = t.rand(5, 3)  \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 3])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(x.size()) # 查看x的形状\n",
    "x.size()[1], x.size(1) # 查看列的个数, 两种写法等价"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.Size` 是tuple对象的子类，因此它支持tuple的所有操作，如x.size()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3989, 0.5617, 0.9267],\n",
       "        [1.4302, 0.4502, 0.5107],\n",
       "        [0.5054, 1.4337, 0.4496],\n",
       "        [1.4012, 1.5774, 0.9716],\n",
       "        [1.5367, 1.2256, 0.5118]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = t.rand(5, 3)\n",
    "# 加法的第一种写法\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3989, 0.5617, 0.9267],\n",
       "        [1.4302, 0.4502, 0.5107],\n",
       "        [0.5054, 1.4337, 0.4496],\n",
       "        [1.4012, 1.5774, 0.9716],\n",
       "        [1.5367, 1.2256, 0.5118]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第二种写法\n",
    "t.add(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3989, 0.5617, 0.9267],\n",
       "        [1.4302, 0.4502, 0.5107],\n",
       "        [0.5054, 1.4337, 0.4496],\n",
       "        [1.4012, 1.5774, 0.9716],\n",
       "        [1.5367, 1.2256, 0.5118]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第三种写法：指定加法结果的输出目标为result\n",
    "result = t.Tensor(5, 3) # 预先分配空间\n",
    "t.add(x, y, out=result) # 输入到result\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "最初y\n",
      "tensor([[0.4018, 0.0661, 0.1942],\n",
      "        [0.9261, 0.1729, 0.3974],\n",
      "        [0.3494, 0.5539, 0.0621],\n",
      "        [0.5915, 0.7290, 0.3228],\n",
      "        [0.9858, 0.6441, 0.4047]])\n",
      "第一种加法，y的结果\n",
      "tensor([[0.4018, 0.0661, 0.1942],\n",
      "        [0.9261, 0.1729, 0.3974],\n",
      "        [0.3494, 0.5539, 0.0621],\n",
      "        [0.5915, 0.7290, 0.3228],\n",
      "        [0.9858, 0.6441, 0.4047]])\n",
      "第二种加法，y的结果\n",
      "tensor([[1.3989, 0.5617, 0.9267],\n",
      "        [1.4302, 0.4502, 0.5107],\n",
      "        [0.5054, 1.4337, 0.4496],\n",
      "        [1.4012, 1.5774, 0.9716],\n",
      "        [1.5367, 1.2256, 0.5118]])\n"
     ]
    }
   ],
   "source": [
    "print('最初y')\n",
    "print(y)\n",
    "\n",
    "print('第一种加法，y的结果')\n",
    "y.add(x) # 普通加法，不改变y的内容\n",
    "print(y)\n",
    "\n",
    "print('第二种加法，y的结果')\n",
    "y.add_(x) # inplace 加法，y变了\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，函数名后面带下划线**`_`** 的函数会修改Tensor本身。例如，`x.add_(y)`和`x.t_()`会改变 `x`，但`x.add(y)`和`x.t()`返回一个新的Tensor， 而`x`不变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.4956, 0.2773, 0.8798, 0.8484, 0.5815])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Tensor的选取操作与Numpy类似\n",
    "x[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor还支持很多操作，包括数学运算、线性代数、选择、切片等等，其接口设计与Numpy极为相似。更详细的使用方法，会在第三章系统讲解。\n",
    "\n",
    "Tensor和Numpy的数组之间的互操作非常容易且快速。对于Tensor不支持的操作，可以先转为Numpy数组处理，之后再转回Tensor。c\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.ones(5) # 新建一个全1的Tensor\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.numpy() # Tensor -> Numpy\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1. 1. 1. 1.]\n",
      "tensor([1., 1., 1., 1., 1.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones(5)\n",
    "b = t.from_numpy(a) # Numpy->Tensor\n",
    "print(a)\n",
    "print(b) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor和numpy对象共享内存，所以他们之间的转换很快，而且几乎不会消耗什么资源。但这也意味着，如果其中一个变了，另外一个也会随之改变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2. 2. 2. 2. 2.]\n",
      "tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "b.add_(1) # 以`_`结尾的函数会修改自身\n",
    "print(a)\n",
    "print(b) # Tensor和Numpy共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你想获取某一个元素的值，可以使用`scalar.item`。 直接`tensor[idx]`得到的还是一个tensor: 一个0-dim 的tensor，一般称为scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2., dtype=torch.float64)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar = b[0]\n",
    "scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar.size() #0-dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar.item() # 使用scalar.item()能从中取出python对象的数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([2]), tensor(2., dtype=torch.float64))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = t.tensor([2]) # 注意和scalar的区别\n",
    "tensor,scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1]), torch.Size([]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor.size(),scalar.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 2.0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 只有一个元素的tensor也可以调用`tensor.item()`\n",
    "tensor.item(), scalar.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外在pytorch中还有一个和`np.array` 很类似的接口: `torch.tensor`, 二者的使用十分类似。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = t.tensor([3,4]) # 新建一个包含 3，4 两个元素的tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar = t.tensor(3)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([3, 4]), tensor([1111,    4]))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_tensor = tensor\n",
    "new_tensor = old_tensor.clone()\n",
    "new_tensor[0] = 1111\n",
    "old_tensor, new_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，`t.tensor()`或者`tensor.clone()`总是会进行数据拷贝，新tensor和原来的数据不再共享内存。所以如果你想共享内存的话，建议使用`torch.from_numpy()`或者`tensor.detach()`来新建一个tensor, 二者共享内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1111,    4]), tensor([1111,    4]))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_tensor = old_tensor.detach()\n",
    "new_tensor[0] = 1111\n",
    "old_tensor, new_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor可通过`.cuda` 方法转为GPU的Tensor，从而享受GPU带来的加速运算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在不支持CUDA的机器下，下一步还是在CPU上运行\n",
    "device = t.device(\"cuda:0\" if t.cuda.is_available() else \"cpu\")\n",
    "x = x.to(device)\n",
    "y = y.to(x.device)\n",
    "z = x+y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，还可以使用`tensor.cuda()` 的方式将tensor拷贝到gpu上，但是这种方式不太推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处可能发现GPU运算的速度并未提升太多，这是因为x和y太小且运算也较为简单，而且将数据从内存转移到显存还需要花费额外的开销。GPU的优势需在大规模数据和复杂运算下才能体现出来。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### autograd: 自动微分\n",
    "\n",
    "深度学习的算法本质上是通过反向传播求导数，而PyTorch的**`autograd`**模块则实现了此功能。在Tensor上的所有操作，autograd都能为它们自动提供微分，避免了手动计算导数的复杂过程。\n",
    " \n",
    "~~`autograd.Variable`是Autograd中的核心类，它简单封装了Tensor，并支持几乎所有Tensor有的操作。Tensor在被封装为Variable之后，可以调用它的`.backward`实现反向传播，自动计算所有梯度~~ ~~Variable的数据结构如图2-6所示。~~\n",
    "\n",
    "\n",
    "![图2-6:Variable的数据结构](imgs/autograd_Variable.svg)\n",
    "\n",
    "  *从0.4起, Variable 正式合并入Tensor, Variable 本来实现的自动微分功能，Tensor就能支持。读者还是可以使用Variable(tensor), 但是这个操作其实什么都没做。建议读者以后直接使用tensor*. \n",
    "  \n",
    "  要想使得Tensor使用autograd功能，只需要设置`tensor.requries_grad=True`. \n",
    "\n",
    "\n",
    "~~Variable主要包含三个属性。~~\n",
    "~~- `data`：保存Variable所包含的Tensor~~\n",
    "~~- `grad`：保存`data`对应的梯度，`grad`也是个Variable，而不是Tensor，它和`data`的形状一样。~~\n",
    "~~- `grad_fn`：指向一个`Function`对象，这个`Function`用来反向传播计算输入的梯度，具体细节会在下一章讲解。~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]], requires_grad=True)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为tensor设置 requires_grad 标识，代表着需要求导数\n",
    "# pytorch 会自动调用autograd 记录操作\n",
    "x = t.ones(2, 2, requires_grad=True)\n",
    "\n",
    "# 上一步等价于\n",
    "# x = t.ones(2,2)\n",
    "# x.requires_grad = True\n",
    "\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4., grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.sum()\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<SumBackward0 at 0x7f63e55b7810>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward() # 反向传播,计算梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# y = x.sum() = (x[0][0] + x[0][1] + x[1][0] + x[1][1])\n",
    "# 每个值的梯度都为1\n",
    "x.grad "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：`grad`在反向传播过程中是累加的(accumulated)，这意味着每一次运行反向传播，梯度都会累加之前的梯度，所以反向传播之前需把梯度清零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 2.],\n",
       "        [2., 2.]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3., 3.],\n",
       "        [3., 3.]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0.],\n",
       "        [0., 0.]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以下划线结束的函数是inplace操作，会修改自身的值，就像add_\n",
    "x.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  神经网络\n",
    "\n",
    "Autograd实现了反向传播功能，但是直接用来写深度学习的代码在很多情况下还是稍显复杂，torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。nn.Module是nn中最重要的类，可把它看成是一个网络的封装，包含网络各层定义以及forward方法，调用forward(input)方法，可返回前向传播的结果。下面就以最早的卷积神经网络：LeNet为例，来看看如何用`nn.Module`实现。LeNet的网络结构如图2-7所示。\n",
    "\n",
    "![图2-7:LeNet网络结构](imgs/nn_lenet.png)\n",
    "\n",
    "这是一个基础的前向传播(feed-forward)网络: 接收输入，经过层层传递运算，得到输出。\n",
    "\n",
    "#### 定义网络\n",
    "\n",
    "定义网络时，需要继承`nn.Module`，并实现它的forward方法，把网络中具有可学习参数的层放在构造函数`__init__`中。如果某一层(如ReLU)不具有可学习的参数，则既可以放在构造函数中，也可以不放，但建议不放在其中，而在forward中使用`nn.functional`代替。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        # nn.Module子类的函数必须在构造函数中执行父类的构造函数\n",
    "        # 下式等价于nn.Module.__init__(self)\n",
    "        super(Net, self).__init__()\n",
    "        \n",
    "        # 卷积层 '1'表示输入图片为单通道, '6'表示输出通道数，'5'表示卷积核为5*5\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5) \n",
    "        # 卷积层\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5) \n",
    "        # 仿射层/全连接层，y = Wx + b\n",
    "        self.fc1   = nn.Linear(16*5*5, 120) \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        # 卷积 -> 激活 -> 池化 \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        # reshape，‘-1’表示自适应\n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只要在nn.Module的子类中定义了forward函数，backward函数就会自动被实现(利用`autograd`)。在`forward` 函数中可使用任何tensor支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。\n",
    "\n",
    "网络的可学习参数通过`net.parameters()`返回，`net.named_parameters`可同时返回可学习的参数及名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.weight : torch.Size([6, 1, 5, 5])\n",
      "conv1.bias : torch.Size([6])\n",
      "conv2.weight : torch.Size([16, 6, 5, 5])\n",
      "conv2.bias : torch.Size([16])\n",
      "fc1.weight : torch.Size([120, 400])\n",
      "fc1.bias : torch.Size([120])\n",
      "fc2.weight : torch.Size([84, 120])\n",
      "fc2.bias : torch.Size([84])\n",
      "fc3.weight : torch.Size([10, 84])\n",
      "fc3.bias : torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "for name,parameters in net.named_parameters():\n",
    "    print(name,':',parameters.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "forward函数的输入和输出都是Tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.randn(1, 1, 32, 32)\n",
    "out = net(input)\n",
    "out.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad() # 所有参数的梯度清零\n",
    "out.backward(t.ones(1,10)) # 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，torch.nn只支持mini-batches，不支持一次只输入一个样本，即一次必须是一个batch。但如果只想输入一个样本，则用 `input.unsqueeze(0)`将batch_size设为１。例如 `nn.Conv2d` 输入必须是4维的，形如$nSamples \\times nChannels \\times Height \\times Width$。可将nSample设为1，即$1 \\times nChannels \\times Height \\times Width$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 损失函数\n",
    "\n",
    "nn实现了神经网络中大多数的损失函数，例如nn.MSELoss用来计算均方误差，nn.CrossEntropyLoss用来计算交叉熵损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(28.6152, grad_fn=<MseLossBackward>)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = net(input)\n",
    "target = t.arange(0,10).view(1,10).float() \n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(output, target)\n",
    "loss # loss是个scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果对loss进行反向传播溯源(使用`gradfn`属性)，可看到它的计算图如下：\n",
    "\n",
    "```\n",
    "input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d  \n",
    "      -> view -> linear -> relu -> linear -> relu -> linear \n",
    "      -> MSELoss\n",
    "      -> loss\n",
    "```\n",
    "\n",
    "当调用`loss.backward()`时，该图会动态生成并自动微分，也即会自动计算图中参数(Parameter)的导数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "反向传播之前 conv1.bias的梯度\n",
      "tensor([0., 0., 0., 0., 0., 0.])\n",
      "反向传播之后 conv1.bias的梯度\n",
      "tensor([ 0.1366,  0.0885, -0.0036,  0.1410,  0.0144,  0.0562])\n"
     ]
    }
   ],
   "source": [
    "# 运行.backward，观察调用之前和调用之后的grad\n",
    "net.zero_grad() # 把net中所有可学习参数的梯度清零\n",
    "print('反向传播之前 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)\n",
    "loss.backward()\n",
    "print('反向传播之后 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 优化器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播计算完所有参数的梯度后，还需要使用优化方法来更新网络的权重和参数，例如随机梯度下降法(SGD)的更新策略如下：\n",
    "```\n",
    "weight = weight - learning_rate * gradient\n",
    "```\n",
    "\n",
    "手动实现如下：\n",
    "\n",
    "```python\n",
    "learning_rate = 0.01\n",
    "for f in net.parameters():\n",
    "    f.data.sub_(f.grad.data * learning_rate)# inplace 减法\n",
    "```\n",
    "\n",
    "`torch.optim`中实现了深度学习中绝大多数的优化方法，例如RMSProp、Adam、SGD等，更便于使用，因此大多数时候并不需要手动写上述代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "#新建一个优化器，指定要调整的参数和学习率\n",
    "optimizer = optim.SGD(net.parameters(), lr = 0.01)\n",
    "\n",
    "# 在训练过程中\n",
    "# 先梯度清零(与net.zero_grad()效果一样)\n",
    "optimizer.zero_grad() \n",
    "\n",
    "# 计算损失\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "\n",
    "#反向传播\n",
    "loss.backward()\n",
    "\n",
    "#更新参数\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "####  数据加载与预处理\n",
    "\n",
    "在深度学习中数据加载及预处理是非常复杂繁琐的，但PyTorch提供了一些可极大简化和加快数据处理流程的工具。同时，对于常用的数据集，PyTorch也提供了封装好的接口供用户快速调用，这些数据集主要保存在torchvison中。\n",
    "\n",
    "`torchvision`实现了常用的图像数据加载功能，例如Imagenet、CIFAR10、MNIST等，以及常用的数据转换操作，这极大地方便了数据加载，并且代码具有可重用性。\n",
    "\n",
    "\n",
    "### 小试牛刀：CIFAR-10分类\n",
    "\n",
    "下面我们来尝试实现对CIFAR-10数据集的分类，步骤如下: \n",
    "\n",
    "1. 使用torchvision加载并预处理CIFAR-10数据集\n",
    "2. 定义网络\n",
    "3. 定义损失函数和优化器\n",
    "4. 训练网络并更新网络参数\n",
    "5. 测试网络\n",
    "\n",
    "####   CIFAR-10数据加载及预处理\n",
    "\n",
    "CIFAR-10[^3]是一个常用的彩色图片数据集，它有10个类别: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'。每张图片都是$3\\times32\\times32$，也即3-通道彩色图片，分辨率为$32\\times32$。\n",
    "\n",
    "[^3]: http://www.cs.toronto.edu/~kriz/cifar.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision as tv\n",
    "import torchvision.transforms as transforms\n",
    "from torchvision.transforms import ToPILImage\n",
    "show = ToPILImage() # 可以把Tensor转成Image，方便可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /home/cy/tmp/data/cifar-10-python.tar.gz\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# 第一次运行程序torchvision会自动下载CIFAR-10数据集，\n",
    "# 大约100M，需花费一定的时间，\n",
    "# 如果已经下载有CIFAR-10，可通过root参数指定\n",
    "\n",
    "# 定义对数据的预处理\n",
    "transform = transforms.Compose([\n",
    "        transforms.ToTensor(), # 转为Tensor\n",
    "        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # 归一化\n",
    "                             ])\n",
    "\n",
    "# 训练集\n",
    "trainset = tv.datasets.CIFAR10(\n",
    "                    root='/home/cy/tmp/data/', \n",
    "                    train=True, \n",
    "                    download=True,\n",
    "                    transform=transform)\n",
    "\n",
    "trainloader = t.utils.data.DataLoader(\n",
    "                    trainset, \n",
    "                    batch_size=4,\n",
    "                    shuffle=True, \n",
    "                    num_workers=2)\n",
    "\n",
    "# 测试集\n",
    "testset = tv.datasets.CIFAR10(\n",
    "                    '/home/cy/tmp/data/',\n",
    "                    train=False, \n",
    "                    download=True, \n",
    "                    transform=transform)\n",
    "\n",
    "testloader = t.utils.data.DataLoader(\n",
    "                    testset,\n",
    "                    batch_size=4, \n",
    "                    shuffle=False,\n",
    "                    num_workers=2)\n",
    "\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset对象是一个数据集，可以按下标访问，返回形如(data, label)的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ship\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAIAAAD/gAIDAAALVElEQVR4nO1cW3MVxxGe2d1zk46E\nhEASlpCEBaEo43K5UqmUKz8jpCo/MQ/Jj0j5JQllLGIw2NxsK4iLERJH17PXPEz316OZxdLoeb4X\nWruzvbPDfNOX6Tn64cuRUkopVde1OomqEbmsaqcZhIKbFTVJVVV5jRsWRGdRlaRc8d2Gbmtu3/AD\nTdM4Ql4m0tXavYs+NI1mVepjX9pUckUXlXMX7RMVcWbEwQpAppkCEACttMgsJiw1uOK1EYHvJWht\nvQWqUhY0s0FrJqbGYy5V00S6Bwjf5RpdSZKU/vaorRrpldau2oRfFGdWAOJgBSBLZMLy5OS/Ey1D\nCakRXqAZBDZJunEayRVrkguNEpe3iSwO3DkxWCCv9R0ed1J0hvsO9uFtYLTSNg3d5QhsjTMrAHGw\nApBZfHJnvj2zwYtaYTLz5GzQ5qQiS2w8U6tsHmm3WeL1wmK9e+VEJ+C7ijlkN5VvZQluJM5HqTbv\nF0Y6zqwAxMEKQBysAGRwWH0XO7GMqCxVWNr4AawFvivQ1O6CaLvdWpNLXXNEnYgn7boC0r0Ga6ul\nSvrAjgVPg6pkj58vQUOt8S2W68APIhiIHvx5EAcrAJk18wkWc+ygF3EsLnLuSey9qwNudC0+hJU5\nEp/Ddf0tEvqeDFwHiztYHBpXgzg0csv3VERV43EzevDnQRysAEg+S8JgvldbbEgSmBKka+mWzHyh\nFWJyNIJOC4hs4VJLPotttJDPzZbZy0cj7V2zqFJXuZh7lmp7zqAPSMYpV4g4HXGwAiCBdIt32tjW\nENlk7d/FE87f2mOHnSxqGtCQA1rtPqi8KxBOmjD0wPOQvTy45fm2JA1ASY3eeHyMOB1xsAKQiY/n\n08q68BteX4vQss9DQm3lfxEb4lErGhVb62mSXLXVP5ek1h0v2e0pP/HtkuJ2I9w4swIQBysAWcPT\nrvIqAM6I1CcdiMIzuYBHmGR4MEFWl1mQ8pNlUzhv0QolCOzxCotV3fD/OvIwKLNgVbVmd9oLRWtR\nLouD1q6vHGdWAOJgBUBI0VKMcDYknt8olUaNn8axd114ejeunRKC1O5mkrWDeyI6dL7DN82+oYQ1\nTKz8rV8A4Wd7Ik5HHKwAZFq5+/26JegTaEmQIg/jjnjLhEey0dr8hJXBbgjebO3XukkebM3aqkBk\nLQkc3PXKJrxMqd3hunYXkxQ+s/tVER9HHKwAZLLm+5nEVrRUyzGJavdOW+7UripgS4dSPNGIJCZt\nKaZI47CG1OonzC7KpPx9B+isGrwOHRUaViBpTZ5qmqb8FRFnRhysAMTBCoCUSYK24H9b4liWNvHF\nUZXfkrjlhYP5n1mrQ8aBcCUmPOFuUbMclf68q4QOpFYVJ+JorINwgLAdlXiOQtW6THsZ7RhInwdx\nsAJgbbJaRbzm38r2tgE/Nhan2fWMMZNB0IP9D9C0vf3OCEXB2StW1ZuYcl47nBxSr/hQTpL15TO4\n82VJroZfoCDuiFecUdvBAF/WnBqLZ3fOgzhYAcjatm1cwYb2JrNVe8R/s+1Dk4QLfp/98BCq7t69\na4TxeGyEPCc+Fg1Zyi++/NIIn9++bQTQcHK2B1U4QqekNApW3k1tV6UbFdi5A1hPmGZvszXiDIiD\nFYAsscp86N/W3BPguay1xmRmTR5/Gy6xXbh0ERdXlz+hFzEdtt+/N0JeEw0zVvr4+wdGuH79Bt86\n8Qb+CPSKbTrTFoF3guJcvlLZpcbMOkmXt5Q2RpyGOFgBaNndCd/fYQ1ylo6Jyf8X+TGZuV5X3njz\nxroRpqbIBf3mm3tG6A5njXBwdER9YtZfnL3g99M6g4cKRWTZvEIoT0r8PLhStXcsPs6sAMTBCoCY\nlMqLqqQw1vb6pJCBnTdVOQ+CAjjR8fbtKyN8d/9b6Dw+PjbC5i+/GCHNiKTXrpOw9XLLCF999Sfu\nFPWqKqQeIvUOi9f8OR22ffiZCvldB8mQW7UOqPzDOHBqO86sAMTBCkBWeT+XIlV6lt2QX3GQ/U9q\nX1aF00YOjLEvOneZrJvqiDVMFQV3U3Nz1GyOXNa8yo2w9YpoOL+wyMq5JMi22rUwivopd9wtnFq5\nYeOJPSfv5EyTRGsYjjhYAcgQOllzklDVYiPQLFMwgkix8hlLsaL0f3BhetoIPzx5YoT5K8vQeXBw\nYISpGaLh/v6+EV5vEfuevPjJCH/7+z+M8Jc7fzVCryuZUuvnlOhKXoBE2hFg2cUVtew+fNESzWKt\nwzkQBysAcbACkB0XpXNJ9kUsM4/cccXubJmT/52mXW5BQ//zTz8b4e3bX42wf3hohPxEJRScD96w\n6Q2MsLh01QhXr103wmBIy193YpJ7YvWZ/Ymyoe6N+St6aYe/y1udJeQQVVhwk9oNSOLMCkAcrABk\n9+7/10jwtuEldKzcU6/DfnNN/vrkgPzvJCEaNglduXdvwwgbG/eNsLu3Z4SF1TXoXF4mN+Lp06dG\nmGNXfmVlxQjrN24aYW2Nkl9vft02wrgQHoJZ45w2ipBTyziQxg6TtfdLRCtKey1q4SZpcC9EfBxx\nsAKQvf+wa6TBgCxRxkmlzLKGmoPJNSbIzDTlgvsDqkJ49uJ/dGuGMr/r69eMsDMi13x6fhE6//Xv\n/xhhc3PTCCWnqO7c+bMRZmcptH786LER3rwmGua2OWQTdshmt9MhIwinPpX9Hg6k4dNbNMTeKtYl\nv4Y64nTEwQpABpNSHNAEnp2l3FOv30W7hUt0scPcHI12jbC3T/Gw4jNqv7tJlmtpiUi3u0c03DnM\nofOPf/i9Eb74/DNqtks6+/zqmRnyRY8OaJvnYH/EfWeiWdVRiIgrzohhdwe0bbyAv2yj4W9UL0Wc\njjhYAcgSnszb22Rl9njCPzvaQbseVwpcmiVepFLaQCPe53I9mNGq5NxQ2bJBsrJ8hVRxVT4MMRzj\nfEz28ZPFy0bY3KRUV29yILqYUKMRkTTPmYZcnIsMV8qVvzCCRdFCQ+tcbsxnhSMOVgCyhmfdxUs0\nz1EOW42lWLbhY9mDASVzUQePCp5KUZuDQ7KPBVfyjXMOPGsxYTnzGDSE3cmYKSknWLocga6vXnUe\nV0qV7HlWnDhqeM8JDNOpe1K8kjNDkjgqeenAmlDHFM05EAcrABkog1mHdAccQqWULjkvyns5OVfN\n9jPKzHSEO8je8OOY+aX1Yww1NjvlPdyM+ctv2d+jDmRMzP60dC/nOG5+boaUF2TT9yoUPXT4HbKB\nRVcSoXQxphdVXAQMWxlnVgDiYAUgO2YaznEyBDwBv5RSyyuU1ex1aTI/evS9EV5uvTHCYEhbCUh4\ndlLyG3WXnUxl5yS50LxyDWuGA6mcGtIDEsbwNot9UcQBYMo1VDOTE0Y4PqRDL3VO2VosF3ND3h9Z\nmIcq1Dq8eU0PVtXgRHcjzoI4WAGIgxWAbOEy0fWIyzQS9iFu3/4M7VaWKTO1NyLmT0xQNvnwmIz0\n0xfPjfDkx2eknVUhRzbJJ+GU5a9P8PrS4aieM2MSig/6tHCguPKoOIYq/KbTaIeC//l5itKHvJIO\np+gtV68sGGHpCn17t2M5NLwX++7dB/5k+sA4swIQBysAGfI+MMljrtPf2JDK4offkYBULJJWq2tr\nRrh165YRUGb14AEduHn+nBi6s7MLnb0eu/68EwNh0KFb3Q7Fz91u12lTWbWNSUqdQeHFCgf8K4ur\nRri6St7PBU6E9bFzbKnCNm2vR+m50ZAS7nFmBSAOVgAyJGum+QDN+JBouPVqE+0O93aNAIp1mBf/\n/PprI3Q9WoE7S0tLRsjzH6ETaazhkExkxldqjl1hm0bcAcTkCJ6VUkfHtIZ8yiVKO2wWYaw7XVI+\n9SkRM0mQ/hYavt+mF/X7ZD3n5siUx5kVgDhYAfg/pQ4eZ65sAxcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=100x100 at 0x7F63D5C372D0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(data, label) = trainset[100]\n",
    "print(classes[label])\n",
    "\n",
    "# (data + 1) / 2是为了还原被归一化的数据\n",
    "show((data + 1) / 2).resize((100, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataloader是一个可迭代的对象，它将dataset返回的每一条数据拼接成一个batch，并提供多线程加速优化和数据打乱等操作。当程序对dataset的所有数据遍历完一遍之后，相应的对Dataloader也完成了一次迭代。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       bird       truck       truck         car\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAAy/klEQVR4nO19Z49kZ3beuaFydXd1\n7p7YPZE5h+VGLtciZThIKywg2DJgwIYN2foB/mL4i/6BYQgwDDh8sAxZa8lrUitpd7XkcrlMwxkO\nh2mGE7o5oXN1V3VXrhv84X2eUzUVhpQNhZLe82HmdNWte9/3vW/VPc8JzxGxYsWKFStWrFixYsWK\nFStWrFixYsWKFStWrFixYsXK3y5xBrzE13w/YZQouut1x4m+8Lxx/P94dRFx9EoS9yh6Wj3ky13o\nrqs5fdfVcziuHuMZJWhjslHU7v7I/b/9Y6O0nBRfC81/bsyzxPp/3DNsEa6hc9eRftz7fihuzyCz\nPpTf+voxo/zyqTEcHOECe/s1EWm18KFMNmmUfA4fTvg40vMwU9eF0gxx7XYbMzpo4zyfH7RE5OJW\nxfz57s0do9yp47OXfutZuVv+zX/9AKeNMBEnxEp6iaSIhFxqnWIU9W6wOO596wtf6fqTm4eL03ml\n7yQ8RKK+wZiDo1A/23u2zi7lK07f2FT5wW//0sAJBkHQ88o9RL8p/cow+TKnjXXYnX2L04Zh3H0S\n/Sq1OexWq4VX2rjLleqBUZp8K5fNGiU/Ntb9SsLH5tQ92T8dV6xYsWJlRMTvf2l+fs4oy8vLRomi\nsPuAmE9/p8towVvR0Iekiv5qxuJ0n0R/+YM2fq1dD2+5nd/VXuNLn2+uowe7IhJGUc8H1AJKpWBu\n5HLZnvMHYcMo1SqUG9fXjLK5WeyexUIBhtVmBacNdAB9U3fUcpPe53PPE8TTecV8yMS4R26M847R\nSkrSSpIQY/C9FI9Ji0g7gZVMJWAsJzkSvZBwAYM2HoAtjl/vocPRmjH5NLETCQzSCwdsJH4WB6eS\nGb6CMbSDUEQcrr7LsbmuGi9qofRaEJ2HfOeVqPuVrgPUetKT6CL3Wr4dpc+4MOfvmPZcHJdnU3sk\ndno/q9+gL7RvfH/oSv5Fi44t4JZQK0kNqGajKSKNBl5v1PE12T+AGVWrVo0S8nR17qtms2mUqckp\no8zP4xgvkRaRdEpt7aFiLSwrVqyMjNgfLCtWrIyMDLA/MxmY7oVJuMQajYp0OWVVCQIYuk4HjvUb\n873uRjV6HceTLg+fgjg1RPW0vk/o4fV6oKMw6jnYHKPOS7V0fRdKKokBTM/ANE3ylXYLBu1BpWaU\n9Ts7MkgeOHYE172NA3ZqsHgloTivF/fFHT/83davAg3iF5eu6BQ/kSfQyDiY2t72hlEqhRmjTE7n\njJLzXRFpc93CABcIAi4gcVlIOMmbKc0OXFWwxlsfhyLixgCYE/mCUTbbDRkiYQNgITeGQbZ5aTcK\nRSTsIFDOnavhRh7f6j2tHqz3PeBonbuPdvXPPn+Fo7A4oGPB9XsOVvxoAPEgH/9QF34nRhTpsn9B\nwOoeTvF+FPzl5R6nVVxcbwC7bW1tGaVcLhulUa8b5aBSFZE6t3qtjq+JgYrS9e3O0LMecHN3jkmk\njZLO4uuWz9dFJJ+BQ8P3hhpS1sKyYsXKyIj9wbJixcrIyABIGIRAHO027EDfRNU6URjFZQE/wrAR\ngYyGfiIijk5iSwfORSIS6AFhr7GtUcKAQRbHVTip4RzNGREqroh4jKC1WgCYCZrlzSrmVXaB5nJ5\noOBmHfjF9/CKO8T8PqjhyETS5fkT/MjQx8AXZ8EQgHhcppMTQHn3z+aNcmMLhvq1z5DitCSIYD49\ntYDBOK6I1Lm2Bw3cqSqTyRp8y+Wd0iVVyKaxxZBrcBA6IrJPpLbXwErWuWf6JZUfh5bg1IK7NonP\nAahzQOF8GPfiPo3Sef2v6Aa7+5Z1goP0CWjQ0+GWizhlz9UMIHxc18cAPUXQXUHJ3n3blSSoQ+Jg\n4y9Ac/1wrx9p3iNNqf9Tw16vE83t7u0bZWevZJTNzU2j1Gq4v+22ZleFIiKcRWc16DSICADrdQDA\ndkAPD8eQ7IrKm/8btZqItFtAkX4mPWxe1sKyYsXKyIj9wbJixcrIyABI2Alx0eBMJdPSZf5pxYbG\n7HxfYUVvNETf6jeYwzCQ7jqAWMOFPD+H5zFupCEbDUe6nStySn5SRCLWfyRp5KsdGxG3Nhq8kAvz\ntVFn1Ugnf2+w1V0lZEukMIBpIGdpBXohXplKVxVH3P2WvuwTm6RjjOQ0Y7VPHC4YJVvbNkp5BSUy\n6/uwpdfWAHJnZqZEpFrHBLeqmOBOFa8oiPMYSYxo1e83eyM+bgrhm3IgInKjhOtuNRAcDP2hT778\nDPKQW7yiL3etraZZxq7iMqIwD2+1WvQbaICvg2T1XIROnidd+8Ht1JFIz0kilynKOiBup4DujnoN\nkzWrkUimeJK7oKIMLPTRrGa395gvL11nYxS1D8UPk/6Kn729PaN8/vnnRtnc3sUx0rekHu5Uowr8\n2G4FQq+L3AWQeRNZDaZJzYUc8N3MNIPyulsYXI6ademKRWYsJLRixcrfABlgYXXspgQeJpWK8fKq\nmaMlFL2GVb/ob7Aeo0lVrpsQEdfBAS065yKW2gZUUil4fz063dVVLyxG9emJjKNQugw3fcC2A7XB\nYAo19dl1gF/6zqOjxchDMDhxZoJpJmqqpFgrvk9zps2xhVy6qM+cRGaW1n/QPXs6jbM9NgtlcfNT\nzP36a0aZmoU/eyuHGRW3bmNqiZSI1Pm4W63BaVphlXJLy1mZHdNysLY7TbxVaTOzzMOtaUaxiGzW\nMOWItyPlDS2qSCUwts4WuXuzDEhf4lZRX3WnsokWqBaBaR2JmqkuzsbFp8GupSFBC0/ysIbwRdiA\nGZUSLFSVPukqQwR+fkpEZheP94wkinpvrr6imzCidf//Y2GpfJk8LHMZ/fZtb8Mqv3LlilE02cpP\nYycn0nl+WL8ytKA70YlIuoBCV6Yky8V4u2cnJ4yyNA/DKgywtq/+9FWjtFq4I7/04osi0iogvhRH\nDNT0ibWwrFixMjJif7CsWLEyMnKv0nC1O00tTpchOpSFp7+qJkET0aMDr8vL7ksX65bjQPHpCtVi\nHd9P8RiCUPqko5j8QVp4EUbS7ZVP6EcIZOLeQQaEE61mi1NODliRLjk1CaRwvUx/MCc4liLtAWFr\nwEKckNQLncp+M0eONkeH+iNN5HlNrV41SmoVWVcLFWRdZdModhm/Ck/qwdSsUa6mpkRkK4bDfotD\naTAhq9FxRfeyTdQCdTxrZQ9d9YZII0FSLa3E8oZupMQXLGSH9qCD7bRihgs4KDuJ6VdKa0GXbb24\nKiIJhw5duhqOIrVOxnO4QI438dAsYEsmjVe+/5N3MP4xYMCWZ0JPTOtLwjHsEAC6HYas3kF2cXL1\nTn+Y9CPBTo5YH8GJKsqRELUDEdktIgjz2dXPjLK+gXKuFB3bmRyAmHowggBIzRNMNk8nuOunRcTh\ndypJV0CW1TaJGMp4FgAzbOK+eLp0LQDwX/wMpHJnlqdFZGEOX/N2beimsRaWFStWRkbsD5YVK1ZG\nRgZY8lGnxhz/G3ilJA36hsYgVNGigQQjjAkiPj2mFSmPQktEPE8h4V2Xk7sKfXghBho0eSdJ0i+X\naCuMA+kKw0UayuRcPU6wi6hAqeqYBpVWJDs4/jXXummUuouZ7gdQJmamjdJkCUuJ+VBtYqs4VEjr\nSldcbHH9jlEu/+IHGFsOq3FkGjZ29fCSUU4fheJfX8HwNxAlvDPXEJEVF9Z+nda+qzliCmSYLhZQ\nacdcbVchMyXhi4jP55zPLREPh4QavXWSuNFxTw6a3qk+SoPY06wrjbLhbGm9m+RcHMvjmMWZpIi4\nIXDNVpO3kptHyQAO9hElvLoClL1RBLhe20XByuzZSaNkvbyIRIQzY3mMoK7bmds1DnuDg/2RxGHS\nz27SRYXSa17oN1UJNjVmurWzLSJXPv3E/Fks0o0wjl2UJilLu1MbB0UDfDlG53c2141SqVdFxOet\nnMxjz+cYJdxcRVlPpoB1WTiGRLyVyx/zGHg56rvAp+/+/E9FJOnjS/GNF/6BDBFrYVmxYmVkxP5g\nWbFiZWRkgCWv4TaNDaGEYnjVuNuh5VYAhVeULKFTOuNpyC8WkYDgSAHagKYpOiQFC9q8hBGrDKkF\nPEmKiBBpzh06bJSAbW/Wbl7HK8xbU8rtZELxaW+ZQo9c+F//HgdkYD8ncojZRfO4YjJDNEEiByeL\nY4JEnuPPmA+bP5s7MJLfe+89o8w8/YhRbsWANoUZHHzfIbxV5CLLFQSDkkFdRA6PY15pBnEyROgJ\nYuqZAvIGQ075vTuARXfI7dDkYorri0iCDIKO3oXhjGsaSNKbGikGxL+9uaB6QBxrmJDkhQShYxEG\n2SwBX2zvIyK2Vd8TkSJRzO0NvF6qYNjJHDB7WEFMdotvuRm85fGOnMhA8VJ5ESmu4SNRBQAzO75o\nlHak3yZlKOHffeU7XygDjhyeLqoHV2rId129fUtEigfAvNkJxIsT3OEBh5Qk3J5lEVizBpR97r0L\neIUtDpZOLIvIDBdwdhyL4/MGTZ5FzmeGTqEL7/zCKG/8BDHBG8SGh2fxddjd3BKRt8+dN3+euL+3\n95KKtbCsWLEyMmJ/sKxYsTIyMoit4W5ydBEJo1gGdSjSEnC1M7u4+XoL1uvM6+swOkSeiDSZq6nl\nSKkUM/w6Z9MsR+WHhumr9YZeDvgl5edExE8CqeUnDhll/jCiFfsl2MmVJlp4aem/dJCghqV6Z21k\nLEA0JLmPuJ5fwlvBLWZpugBiTYHlHDA51uMc/bFZEQl8HPDppwg+OpMAGounThjl5mWEAo96rHxM\n804xiTHHW/bM4XERmTlBcEqLPWD2bHUf2HN3C6fdJYF3cg0zccq4UIKwNzk+JSJRHn+mkxi2O7xB\nU4Zl+2FnSfmKiHRzWvR1PFX+AK3+axyAWuD9iz8zyspnwBFZnuiTa7dFpEKCgbAJ4vBpLQNMkRVA\noBw7A3A9d/xBXDHEpfMTyCn1vaSIHLAQtbSH0y7OAPtocFCZArWGVDu//flLCbsqdodjQmXau3Mb\nu3G3VBKRBLNDHeXk6GutkOV92bq5apRX/+x1o2xvAncvL50yyv1nHxKRPPFy4wCLMDmB/RzGcLO8\n+867RvmTVxDvLt66ZZQF9lIYLwCE3i6ui8idDcDtUqU6bKbWwrJixcrIyCC2Bv6OJ9RdGiaEdpZ0\n5Ulpx9Ogpc8WcjMwMWefP5alEhJbfD6jjPdda3f8FLy/yTQz9MmUoENKdup7+Egn2YP2h0+NjYvI\n8un7zZ/FffIZ8bMTfGbWy/hF9+lFjvkQ6yoYGvxYW2trIx8ckGHFT1YpH9T5GmERInrHI/qzw4NN\nEQlrnCm7tT795GNGOcLH0eYYjK/nTsL4cttY0kk+0uMYT9rqrXdFZKcCzqOVTdhT1zfxBF5bu2GU\nLb5VZQvMljbUYWKRSweqm0qLiDsOx2p6HE5Tb3xOhkiSzMhhX4NZz41EREk0mkqdzANzdLoneRM+\nZp7aNqu1Dn3tRaM8Nl7AjH7vd0XkKJOJbu/CcvRdbLCY+0qb5SgrdCKHkxxaOGqUDLvtmiFM0U+8\ns4XNk/L64jPcaVps9OXzsFT6eZD7y3qUs2ST1At37tzhpSMRSZHIjBtECnkYNTub+MgbP3vNKOsr\nsLXTpG14/lvfNMrRYyf4VkpEhIGyKinFS3sAK++9C4P3/fPvG2W/iIW6/8xpo5w4jI5T+6CBEa96\nICJHl3HA4pHjvVOlWAvLihUrIyP2B8uKFSsjIwOd7qook19GRIQudu0sGdOT1ymv50nqzAcplWE0\namOSVBoeO9Nq0ekjxo07RAu9eV5qZHco/QgwHRZ4L505KSL/8l//C/Pn5iYs0kvvXzSK0oZ1MRdr\n6Q+Jj3m2/noII0XOuUU2u+XZeaPMzEFJErZm6RTPpoFTXJ9Ody8tIg4xdS4PAHL0KCDJLBfh0ENw\nfE7nMbj6lXNGSZGq+HYM/PvTP/3PIrJN3oVttrcpEUo3eat0gm3mqXnEL0lm1cUNoMX2QSAi7i7x\nPuFwlBj65Gsqt0QWU1YnQNSuSxdaSTBZzGENzTzLX6oeXjlXhuM2aim7Ls62sgKCw1TaFZFpssGV\nD0pGyZA/oLwNNKQtkWqar0R3xPQ0s+qiu7IRc2OYRXEby+XHDE0QOAfqr+hUHWk1m9xb/lxNUjVg\nVSwiFqGhrXQuJSJJetbHmU22fQfpaS9//w+McvVjlO889fjjRvn2d/6OUU6eOWOUaidiFolIi/VJ\n5y68bZSbNwEn3zsHTpENfu+mJuFAeHoe3wt3DN+C0m7JKHPHz4jICy/9PfPn/OGjw6ZsLSwrVqyM\njNgfLCtWrIyMDICEygHW6vSzdKSLdyGmkayF+A0yoJeZy6Mk75qrlUxmqMByTqcz0p3D1Zej0mGM\n1hogQkINF2qz1YiV/blcTkTmSJmQoFV87hcoEZBOW1ZlLCD28ZXIgWx/Q8I6e0RJ9SrM8geOP2yU\nmZnHjLJyFTVAl7fLnBLgA8nQAUt9oqTlUwig7IWICU6zyWtqnAlBOYTkFvMw75sbCPkxIiflWiwi\nG01crk3w7jNspMFNBf7Ke9Fq6tw1LkwSjrQn3A8ikuZdduMOoUOPVIuIH83kEfrJag+eg5qIZHgX\nJiexE2pFhLoaW/An7B8gHnrfLCDzk+NYKGVEqJWxUFNjUyKychPRzyNzeL1FdskdNsKZnsMmSTNZ\nrJDD2DR9rEPQHsci4iYYFE5wY4TYCZks+UI06yrqrTYLnC/AhP0kDf2i96XCwK5GeDW2mM/mRCRH\nhHv1E+DlH738Q6Pcub5qlNlJbKcnn/yqUZaWT2Ii9PEkcox7hk0ReettfJXevQCawxur7MGzycAf\nw7gRaSE+WsEVd8rwF01Ng2/y2W9+XUQeefwp86c/nPXRWlhWrFgZGbE/WFasWBkZGQAJNWyUZJTL\nJIg62giTBzTasBjL7M4YMfEyTbM/mWIUj281GLHykikRyTNkoLUCDQIZLUXI0LJN8Bc2UiYJbb/K\nLM2sCdYw0vSffuc/GOX7v/c/jfLAfQi3ZdMIS2lwMOZp3Q5v3+Biisk8mBgkAGyZnT1mlEtkg7t+\no2SUUoOptoywRqTNNtR9YyTnyzdx2pX3rhllYRLLMr2wZJRfXIH9/72nkWh3RDuevn8JM0onRKTJ\n29tWcj4SnGvyZCqJG6TYXIj9tfVpF3d43D0LDc16YU6GSKMESOjOImWxQZLvemlLRI4vYsqHx3Ha\ntQMoDdINTmcRaWoUiOLpWGixTdlGBW/VmxkRmVxAhCtHzLi9AZQ9PoXtunBkCXNnpZTDCHLeI98k\nk4ojA12JfSfGOGUuaZbx3HYbey9qs/8oXQAew7g90t8ttT9xVIPwTW6eEoObinZzKYxqvjAhIlev\nfGj+/P7/+O9GiVs427dfeMko2gGsXCkZxaUDYYIBvusr2I1/9Mofisili+/0zLReIZ3DEUD15RNY\n/0weQyrtIpRZYDz97//ad41y6r77RCSdwbfgHkVI1sKyYsXKyMgACytFduBDswWjOO2GiER8Jod0\nQ27t0cFG66ZK3hx1uudz+NUMaMW49Jebrt9aIJpN8ynHA9TK0SKhBM8b8pEStRkBIHnuRC4vIsUN\nOKQvvPWGUTbuIIVncQ6P9CkSAGVYr9BmsZG28BnWI/al5543yss/+hFmymSrdppd5qe4vBG5kyZw\nac9Bmk/VtLHJ4im0mMF0dsZw3RNnkJPiT8CCe/llzGjvAKbcb74Eb2XuDMaf27ghIknWDDscgDJY\n1elZb7JbakOY2aQVOVqb1WkIGouIaG6az/DF8DysqI4HeDYBp/v5j0GHZDqbjiUxQWY4yY1buHel\nfdiSNUZ11vYwo0odo00zeqBl8M12W7rqgR1aNxNj2IoPPoKEIzXqla9qZxNW2CHag4XJSZ62JSJR\nE+7/MZftcjm282+DjOzmGsYfc5eO52GqnD41uIapn2xuUAJgr5m8z+iB1gBN5HChVrUiIq/+CC72\nNq3av/vSrxjl2ae/YZRz597mTBHQmJ1HLGLlBoI5f/j7SNr66IP3ROTQIpzlaX5nkzG4uY8fXzbK\nQw8iBqUJWd4E7tQ3n/+2UR594glM33FExFFK9L6Zq1gLy4oVKyMj9gfLihUrIyODuuYwjUTLa8J6\nWUQCQsIGu3KukS52swhsWCMbwSY9eYuLoGQap3O905UnDkWkSWN1ZhZ2ZpJ2psuqDeUydlgsot73\nOqsEtPtLtFcSkQs/+an5c38PuGlxAeevVeAmz2WU+wGKw4QjbayiuVo9MpUHvtB2qew1KU9mMJLp\nWYCFDC80Mw2ru8JGrUEiKyIO3ajN0qpRLrfhoTw7BdN6jdDmMNuQ+ORI+PH1klGeP3LWKMtj74vI\njRvIvjkgbHHYsijtsptRx5VLVMKDQwJJbWPTbLWkqwBrjIVEoT+UhODY/CJVZhgRxFUbTRH54Y9x\np6bGcLaLH182ygb3VZtg4f6HgeYOL2DuARPifJJl75dLInJnBUlwU1PAdOksKAcSzAxSd3a9jJN8\neO2iUbwWts3RQ/AiZ5MpEWk2sN/Ku9jh+3Rjn78AqHtdaZRDDPuIVlOVp2WQ9LfG6WJkU5owKPv7\ngKWa9pjg3AsFfMs+eO9NEdnZwpEvfPs7RvnWt77OC+EjDz2MPTPGMMIrL4PB6rVXXzPK9hZopu87\nc0pETh3F4m/tlDCvabxy9BgiWn6KcaQJ7G2XvXZm5vBNVEhu+JrdId+1brEWlhUrVkZG7A+WFStW\nRkYGQMI6E1vubJWMEgQN6W4pysBZaqJglGNU1MbbKeKzd+4gfDDOGM3kBKzWersqIu0A4Z56HdZy\nkkk3ok1TCKAc1vf4GjFpwOidIXegt/q5iLzDOnLXgyG9vAzbvl4FCA2YZySAPhLE0BLJ3qhNj3g8\n7dnTyDcpTMCizpWAJo6FOFtyesko59lm8rUPrxilUo9FZK6ArKJv34cjn2M+S8SEsnAKxzw0hwXM\n8IpKNvDBOaCSqbAlIvMChH57H4vsu0q5gbfqOmXWQ3htTWjq7f6Sch0RadbZTYf3zhsb+uSrMBx5\n8xLGpqwJzfKuiJTY43N7HRlbxR0AkKCOsbWUIbIJROa5DLNykwTct2s3VkQkly+YP+PxBZyEuDLP\nwXpMKHMCbIniGqqCLpKLeZV7+9GHHxGROjP1LrP0Z20bcHJ9h6TJC2DlfuAoEo7m8til+TSO+YHc\nJQNigopXCQkPyEe8dgeX1qD23AwQWcx4fXGnLCLPfeV58+dTTwNKx2RHjBnlPPfem0b58NJHRtlY\nR5Qzm2ProFNLRpnMJUSkyUqpgHQR6qtZvYUd3hJsp23WZp0mgd/SiRM9czXT/zJMFdbCsmLFysiI\n/cGyYsXKyMi9uua45MZznVC6ImiaOJ9lDXcnosGTHCP/XJfAFj1Ly7C0VxKRWhP4osxw3nQKAbc0\nWe60asRL4YppVsbHMcosFreBI+QgEJF3N1d4AJvxMC7WwS86XDVF+18ZYqX6LNY/NQ8ShakQ+GiT\njVofWUYS3WvXEDb6nVfQS3K9pF1/REQqddjY566tGuXf/vqvYV5FsM19dhk5jRffuGiUBmHpBE33\nDNMdU1FNRI6wz+X1IsZWarK2ic+qahUgIuEBFvm80Wmmgyr0aJg0XQaY6sSGeWfARjKyxCjbwWeI\n/fnEbrPjYyJSpYtAKQcyrBbqtGXKIoCYG8ME9w5YnsLxa3Cz5WdEJJNDcNBhRyIlz1OWuyTjlRqe\nChmzLrPXTqMKEHqwVxSRA8bBtdyqXsdOaNVx5LGj2MC/8hzCYeMTjGkSRA+TDluDkqa0ALt2toFS\nNT07yVzlPAN8W7eBv3L5nIgsHbkPpyWqn2aLgJs3kRR6/gLa22ytM7J/GJB2eQkbuMRwZKlcEpGQ\nrCDNkFQZ/BYXmSe8y0U+fQZ5At956ZeMMsG4rcY9DZNifK+MUYi1sKxYsTIyYn+wrFixMjIyyJLX\nUvzOK7GIxIyXKTZUXjeFhFq97TIYcYoRgfV1hA9u3QS0efapZ6Sr1GvlNmr9XLJEzM/CdMwREmo0\nZ7bADExBSHGctvrOXk1E1kqAisdmj/aMVgn8IjWUdcpOPxQcbKYWSS5evw7sOeORjYBPgY9Z6vUH\nP38LY/ARIDt2FqOqO5GIlDdK5s/3byCo+h9/+KdG+Xff/WWjVNeBDd96G2GdUg1AUlvPjr0IqzuO\nGyJydBFDevY0rrvOcKFHFFTZZOEbJ1omJXm5pXT7WLGFTF1EmlylDaVVKDLO2idKjn7/fWi8duMq\nSv8bzZaInD2LxEUNMR8+DBCxwQ5RHkvkUgR6a0Vgao+Ib7KgNYNJEUm4uicxkgaTZmvKcF9DCrGG\nWXfYBm1mHGirtMvg49q6iIQMh7k+DpgcKxglT6B08xJ2ePXrGNuhWaRTbu/tyz1FIaHC4VKpZJQ9\nlu6qnaE0+RHh7tWrwN1XPvpURBK8U9MzGOQHF7Gkr732Z5g788MXDyOcOkPYuL2D72y5jGFnM0kR\niZRZk1TxDzyOr3noIJY9u4CzvfjL6BWmPwWdeslOoa4j9ywhVLEWlhUrVkZG7uV0jzo/9qGIpNMJ\nPcT8pzzC2nZUc7Vc0XoC/JpOjOMh/wmL9Vv1poh8/WtfM38+ch8SmpQPa2oa7th52lMp1mFk2OIl\n9JDqcpPP+D956x0RmUkXMFZy3dbaZJLg2MK+Hia+snd1mZcySD787CrmtYeRnPsEhkOT6Uvvvw53\n5nVWfiyfghO0QVM01W6IiD+H+hWWP8nr9L5/sEI3ahZjmxpjoU8CD9i6YP33mjAHtuptEVk8DTvu\ngYWCUQ7xcX04gwEsOfAHf7SGB/j//gB0t7f3cMwYndPffuKUiFxaIZtCBFNlfg7nv4QHfEeuX8PZ\nWvQ3F9iHtVrdF5FyGcEWtbDqNHwc3mXhK7fWkIIUt/HKRBXK5AOPGiWaXhCR8h0YpG//4lWjlGlK\nHGFTlrXVVVyRiUUu4wz7vIkl8kNMH31MpNMktVbGSALSQ5cOMJLtIm73f/ld2MLPP4eb+PjZrHw5\n0fqbHWalaY+cBKNh4+P4goRhk6/gW5bLp0WkSUKuc+eQ93f12me8Aja2rkajWeMVYYVpSZbvs2Ao\nmRKR0489Zv589pnncBK2PnUYflEarAzLktSn/ufqDNQj1sKyYsXKyIj9wbJixcrIyCBIyCoB7VUT\neJ7cVT4uVIitAsWGyn/ApBia1lp8cGiRIO7ObRH5+Rs/N38++uADRjnCA7L0jieIdJwQGOHWFlDJ\np1eBOF5/H2jkwPdFZH4KhT6By9IWbWaiLkO/d/pdRfNKkTzYfC3XMcFiCUjn1Y8uGGV7E2AkyRSw\n8aPg3muzIejT7BBzcqwgIj9dxXSCOcCl1TKcl69exwSffQT5a//4O6DrO0k6wD9+G8zIPtHiRjsU\nkTVAE4noFF9dRSxib4wVLT6W9JU3Qe3wzhpmdCKD+ztP5fW3aiLywQ7QytFlzOtRVgu9Ir2ytIQ8\nLJ/9ZsaZHfbZ9XEReeddtIMN6B73Exp1wU7Y2gBs3CTN8aSPpKTUDsIUMdmTnZl5EXn/A3InfIjz\nhwSYTrVklMIMEo4mCrhQSHbgEgtKwhhIudkoi8jYJPcVPSS1fSCpLDsEJwjVy3U45m/cxDF+qyT3\nlAZjR0W62A9quB0hO+6MM+sqS5IGn3GqM8vw7u+tbYjIlctYhCuXcXPHxxGHOcEkwQpDQzt7wJ79\nNM1z7A38lW98U0S+8q0X8Pos/AkuvybqR4/V8RKri723UVOvu8WW5lixYuVvktgfLCtWrIyMDIwS\nQlEQZ6BTgyTZMXnB0yRCS7GFicJF7bWpDO4KJL0cPpVOp0UkIqf4u2+CX+HHTIo5fgzxi6+SH3qT\nSPDNTy4aZafMaBEpxBYOL4hIrQLYkmYEUwvK+9PHFNu2CRs9zdAaHCSUI3MIx1RLSFr58CrGtrZb\nMsrDRxD7m5uBHb5AG/7RE8hSObk4KSIJViPdOo9QY4vm8kckOM+xs86RBSxgTP68dh5z/IS9Krf3\n6iKyvw8klWKy214J9v/bBNeVInJt9ljwMTmJGZ2dLhjFb2Mxr1QCETkgzJ8dx30P9zZliMzOIQ+L\nG0GqxCBmitq2R4O2TYKgfQGSqhMoTaVQ7FIOMOUbU1jb9B5WbPvHfyIi1y9dxHWZpJZwoewxaDt1\nHDM9+cCTRtndAADfuYWwWtzCBive2RaRdotFNtzhARnWp7lujTq2tJdkal4C+Gu1qt+4n0uXtFmk\nssf9vMvUpya7FqTSLDIjlUjM+qF6DVf89AJmffHdd0Sk2cICLsxibGk2oKpX8C0rMxzZ5vdag48n\nTqKj6gsvgIX94UefEJFUBvNy4166Qa3bc/pAYpdzxel75cuKtbCsWLEyMmJ/sKxYsTIyci9O97DD\nXReLiEeEGJLDTMv6Y/0IU0k9JV9nJK7NGFBMMOLEjohk2M00kYOdWaON/TmzBKc8hEX22LE1dIGG\nlk8CdmVSwAimpkFNXxW3AwkZyCDe01e8TqOhXmqKHtm+heCUW4ax/fgp8pMtIpA0z5S5IokcCjlM\nVvMSPymKiExP4SQTOcxCQeL6HjDChysoBpo5+Sze8nD+x7/2LaM8lcN53jx3XkR+fh4ou7QPOB+2\newO7ysSgizBFKvdjc7gjiy18qiqhiFwsAkTETGV86vCsDJEaj4nIarC7i497bkJECgQgmmNcKiIC\neHgO+KsyC8izO4kN0AoZSs5hJ9QIps6/8rKI1OqAS9PsFaAT3CnyJIRFlW0gwYNtRCFbJGlokgRC\nHE9E6gRQiaSygLABMCk3KgdY7fGZJXx0DJDQJed6jzTYtazMkrIW64e0/R1jp5JJYG83Ocg3Xv2J\nUT65hHjxRMYVkanDCNEqm8IOg491XnHvACdJMdXzH/4qmEKeeAK0f9MzwPWGH0LDfP0M9H/RYi0s\nK1asjIwMsLA6XmYaIKbyJsHnuWY29TutNYMjoBnVJsNvp/mHMv44IiJ0KYrHRjWTbJCt/SGrZCma\nW8Bz1anjucp+NxLLXckjnYywTtZY70x1tB2XYV+CSDzE615q4HlXKyJ7ZX4cT7NCAulRSSb11Fld\nkQhwgRbpp69vbonIicI0B6BNZPHZJrNvInZ7XWdP9s8rdKnSFHrw+JJRvmvqnNib9Ae//38wnSbn\nTjPZ5UNLi5/3q7h36ob/2jFM7c2DbRFp8J7eobmRfnJZhkiDpMapFOwCrbhKzBVEJMHaqddee80o\nGQZzZqcKmKDP/q/MUxsj43aW/c0/+BBFJ8ZmTNJEDcgnlWLEI8m7vE7msja911pynMrDVs27GHaj\n0RSRJA15Ib1ymzv4oI5XJheWMMECdsKVa0iDChk96JGQdm7AlMM0N0+CrMcpjqTJvvBvvf4zo1z+\n6AOjZOiPX5wxl8afOzswrCJ+pw7IZTZ/CKGtX/3ed43yledQcKNfoq4viCdfrr3NX5BYC8uKFSsj\nI/YHy4oVKyMjQ5ltpbvyJgxEJElImCIxq/JJdSzGvn6QqnThL5zfMDpELHavEjuo19/j8BrsjBLS\nxM3S77hPhiw3cRc+7UepmofV3winQ0JE8z7p9RrDvR8hZ9DGrVWj3LpJs78Cq75C3Dd9DCUsGwVc\n6CH2Yb0/UxORz+iZ3uQAQkYe0gRBaywoufnT14zSZgbTBNmO3vwUjU+OnVwSEZcu/MI4EHSzgtXe\nKMFFPc1kq/QYlrRCVHKZ5fsTVUC/z72miIwdx5D2shj2G+m6DJGYy66NeJWKyxFfRCYmsJKfsH/q\n5StQ5ueAlBfm4fQ9yiKnWg0gboce+g8+Ym3WfklE0hkypvHIkPzaGfJe5Mcxd5c7zcngQtOHoCwQ\nFpmdnKALPGBYpkWUp/l9mlkW0A1f34Qvf397TQaJBhw87vAU911COwTT/f/KKz80yqUL4FnLEgme\nYmsoI1v0RVRJS92IsCVOM7fxH/3Gbxjl5GnwVWl8TFOofBLSmdF1amv+0sVaWFasWBkZsT9YVqxY\nGRkZVJoz5BXNsVJEpaZjkqU5WpDtM0ajDW8UiGkGkMQtEQmUp5i/np7iSoYAGyRcLpHAz49IIki0\n2KFt9TwRCcn00BXm0+hkb3CzxmKRmEG6LLmMh1WQJ2fw+vIzqPjfuQZsUlzBpXMegkSVEMO+Wscx\n6TImOx66InKdyVZ3NnFAi3SDc+T2K8xgkb0kLp1MYpAJ4u4JZn6trV0RkfEC7sKDz8Pav3EdFMCp\nAGe7/1EE+AqzOFuljsHkWdRyPmCk78SUiDydwpSDAEDjQuZAhsjGNt7S3rc58hyk/KR08dLNL2Cm\nf/ZTZBWdv/CeURbYVvbFF18yysekgVxn61Mn1Gig6crJJDslHYnYKZb+hEwBuE8cDToXcDA3QJrg\nPZ/LiUhCq9DIVKffiya/DlXmUpXYGrZygBopNzOYwC+kK8BjFE8YZy9ugU7vJ6+/g7l/iniolp01\nGAn99PKqUWbmpkWElCLiM0Hvn3zv143y6BPPGCVHzr82A8ceCQKVr6Ujf1n5VsPEWlhWrFgZGbE/\nWFasWBkZGQAJO3UqjJQFYSBdKWfa4lFrdyLiMt9TcndmSJK8PEHz1Wf+Hki/aMkrnXmCuCyOmPjP\nIVUZt/JYWd5mM1GfnxL/LjtWp6Ngtz8z1tXOoKxgCJjIF/eFFI3ks8y3JHfaodklozTuw2rc+Rwc\nBhVCp/Q4WwQJQJYhq2s7ZCNYIrg7jBTZSXblnF9CyCyRwxU9RbsEI46DC2XH2yIyNsUMzPsQXFt4\nCnUqTU7Q13CUj5kmI2AEzwP8iWJCj/2yiESMzHr62Q7ff680GZ9qMxxcOygZZXN9TURWVkBQ4ZIi\n4zf/1T83SpERwEqVfWe5kktHgR+PHcJCldiI9/KVayJS5wRzTLhtEje1GCmrxwh3TpPXvMW2rKkM\nsltdhvyaZjcSAKbTvYmjrYZ+Fq6AwmFQ+KcbIOfQcNvGJ29Ll2jd2+42AOCNKwj4bq2jCOzKVbQ+\ndQhLD7NdMfv1yOo1pMJ+srIlIodOLJk/f+Of/jOjPPU0kGAXUtbaLL/nrajPP/RXbuD8lQ/AihUr\nVr6s2B8sK1asjIzci9M9yQTR5kFDunr+eJ6GAsmNR0NX0VaTtHxqU7bavUTRJgu0w7TOYyMmFrq8\nkEs73Em5PQcniEYdBqEkDkXEZ4WagjuHCFfT4bRBk9adKf5T2OsOCYvU6ogEJXkhYTpik4mX2RMo\nf0+z2WcUM5LFhQ9bTRGJs5jF9CxASiaJcFJA7oSay0hcA2NLMlfWJwoOFCROiIg0XYyk3kDMUfiR\ndoS3wpDYlkWdjRbecoWt07gIniREpEw2Ai2sS3sDNhLGRudAngEyz2FeZTMvIocWAFKmp1mDyWZr\nMd0FLVIvbG+BL79FfHf9BnIy/+CP0M6r0QxEpEbvgTLtJdiNdWwaADk7vYRFSAAA5qcw2rEccLHe\n/igMRCRgHeV+hRW1/BIlmVPq0A5IkTJkwp3maQaXpvrc6uUyUPD1a2AQzJClYaIAbLvf5AZgI1U3\n5DCTeKVRqYnIE8+ih94Tz4DhQzohcrbw0lBgh1bvrzgUeA+xFpYVK1ZGRgaxNdDK0Mp100i1v4u7\n+rMDWk9N0ipkknj2aoqWEuAqTwBO4tzl2heRiClUHh3qjvQmVcV8bivxVlcNDcoHcF2aUT47sCtr\ns+fo4663amTIg7AjSmHs8CNNDrJGw4TUDB2XqqN9gGhEJBO+iFSqsNfanWFjSNkMWQc4kRTtWZ2I\nw/Y8bVpSjWpdROI0rlvnffESvd5xJUvQEvxmnXeKBk6so3JdEfH4zA94Tyu1oaU5u2xEGrQx5XkW\nEj3+2FMiks+yJKhCCir66as1kn9wT07OsE0LR7tKCulMBlObWVwQkWyI06YnYE+NLaLZZzoHeyeX\nhc2SoNc67qTvMWpEU67WMJENtWqZFqcuaq0G407WXaRhIM8fbCJ4Ccyw0arybDjJ+ETBKNkaQisl\nJspV2F9WKeoCH8qDj9wnIs9/g7wLSlOs6WnKCscx/PU1q7rEWlhWrFgZGbE/WFasWBkZGUSRzHyr\nTnm9l5QuY7LzYa8XX3geE7ISADIKW7QzqdboGEVP6nUo93qRmtr/ndY7fWwQCjlNFolexSfJhEPn\nq5KldT7S6Q+kLXa+ABseVPBGg1OuHcA+d+gm9wkrsnSFuuRgG0vDA21q/VNcnRwBoHr9NdmqWSfS\n5Pj1eRM2dZQ0+H1HurrY1mockhJH613QZpxc5ARRtseoiyLKMI5FxOUOCTm2ZELP2ysPnEWeVERs\npaU5hjtwf19zrJRXAwdkCNlS5EFW/KWb5LGnvmKUoyfQiPf21oaIVEOWzrhwqN/ZxYW2qeyTV6NW\n5wTrJaNUS2C887Nw1afGpkTETzAgoFyVhGO6nWKuj9fBgtxOweD9pPQHOfIUp7OY8uYO8svK+9hg\nba5klUQUGueZWwRk/t73fk1ElpeAgrvcLL0LqPJFXpC/PLkH4bK1sKxYsTIyYn+wrFixMjIyABJ2\n2mmwISjr0RkEUa53mpf9dmaNdOBq3CnplyaAmGNjNUVjPZJZP53EELw3MQH7XANJrT6SbM/3ukfS\nIu2ZVvxobEuH3WEsU7rBGMZ2tVaTQaKBUZ9RTl232QmghnYLF0onCGC5CK2GWvU16YKiMWFRhmHE\nNvvc1Bi8C5tseNPqbfeaY6TMF1dEIsJVvQ0azksnlDKBJSaMMCrBRpoA0tc2LXEgIuUG15zX15BW\nvxyanuJo2SeV2LbeaEkXcXiCkWV9xdMcN8aLG8Ruel8mJhDym50Go8OZMydEJMWsqGoN172+ihqX\nT28h1+nzbbIA7uGYCxfPGaW8i45NC8fvN0rBCUVk6TSqbcbHenkX1AtR5Bdnq4i4Z5tvcdf0imZs\nnbkfl9veRF3XyjWULi2fBNw78+AjRplisyUNZR5lsc7p06eli0e/v0HUiMpoj96KFSt/q8T+YFmx\nYmVkZAAk3NoCD/TuLixnA+I6cbO414f/5Vi9viAK0ZXAxlRGvpKhZfs0a81FtG0qA2FENKVSSURm\n2PqxWESQZWpayyPIKc6yoJZiN3LjNepAHM0mi1ruFk3AjCIcEIasRnKZ6slU0jaRoJbItBUbuCIi\ndSWVJ65UTgvNRw0TCmCZYcsly2dxRY+MCy6GpHhZ46H423eVRJxohXegzdIcR/M1mabrJ5MikmPv\n2xrBnacx4D6psX+qrqTuFgPeo04eMmuzdC/oDDvdA1j1wmqqvCZ/agpxMiddgdH1dWCrz9exsZNM\no/zqw8gp/fw6xrZG3sZD4+BXmJ9D2O6hh4+JyDe+Cjg2PZ7hFHs39h7DeTfWsfdqvGVVrthb/+2u\nj2i8dXYR5Ihff+E7RnniGSR/5iZQujQ1i8Brsi8NWL0cxq+igUuVriZ7f+0SRXVsXj9xIMVaWFas\nWLFixYoVK1asWLFixYoVK1asWLFixYoVK1asWLFixYqVv6byfwGQNmMcbFTFEAAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7F632C0DBD90>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next() # 返回4张图片及标签\n",
    "print(' '.join('%11s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid((images+1)/2)).resize((400,100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   定义网络\n",
    "\n",
    "拷贝上面的LeNet网络，修改self.conv1第一个参数为3通道，因CIFAR-10是3通道彩图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5) \n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)  \n",
    "        self.fc1   = nn.Linear(16*5*5, 120)  \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) \n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  定义损失函数和优化器(loss和optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import optim\n",
    "criterion = nn.CrossEntropyLoss() # 交叉熵损失函数\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###   训练网络\n",
    "\n",
    "所有网络的训练流程都是类似的，不断地执行如下流程：\n",
    "\n",
    "- 输入数据\n",
    "- 前向传播+反向传播\n",
    "- 更新参数\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  2000] loss: 2.157\n",
      "[1,  4000] loss: 1.815\n",
      "[1,  6000] loss: 1.676\n",
      "[1,  8000] loss: 1.599\n",
      "[1, 10000] loss: 1.557\n",
      "[1, 12000] loss: 1.466\n",
      "[2,  2000] loss: 1.410\n",
      "[2,  4000] loss: 1.381\n",
      "[2,  6000] loss: 1.331\n",
      "[2,  8000] loss: 1.312\n",
      "[2, 10000] loss: 1.281\n",
      "[2, 12000] loss: 1.249\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "t.set_num_threads(8)\n",
    "for epoch in range(2):  \n",
    "    \n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        \n",
    "        # 输入数据\n",
    "        inputs, labels = data\n",
    "        \n",
    "        # 梯度清零\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # forward + backward \n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()   \n",
    "        \n",
    "        # 更新参数 \n",
    "        optimizer.step()\n",
    "        \n",
    "        # 打印log信息\n",
    "        # loss 是一个scalar,需要使用loss.item()来获取数值，不能使用loss[0]\n",
    "        running_loss += loss.item()\n",
    "        if i % 2000 == 1999: # 每2000个batch打印一下训练状态\n",
    "            print('[%d, %5d] loss: %.3f' \\\n",
    "                  % (epoch+1, i+1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处仅训练了2个epoch（遍历完一遍数据集称为一个epoch），来看看网络有没有效果。将测试图片输入到网络中，计算它的label，然后与实际的label进行比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "实际的label:       cat     ship     ship    plane\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA0bklEQVR4nO19WZMc6XXdqcysvbq6\nem/0ABgAg2UwxAyHo5mhRIkSJdohWrbssOWwFXaEIxzhFz/4wb9DP8ARDlNW2H6wZYclh+RNFmmS\nokmKnN2zYkCgATS6G71UV1fXmpWLH/KcW9Xd1SNRCke45e8+ALezsjK//PKrzHvuci7gxIkTJ06c\nOHHixIkTJ06cOHHixIkTJ06cOHHixImT/78kd3rTb/3TV/RZkimFIACQ87zszzAcZkqUjLhDoZAp\nccKvpEnKg3hxpujbSKOqjh8DyBcG2Z8+An0l1dGiTBlFPGyS2IADjYFbhlJy3DPR0XIaNkcbRzqR\nLtADBxnqWx2eGb2QH/3Gv7uPCdnf3+cAIu6ay02ZzJ9UfrKDpCeV8QYv+5MbvNQ7uUdO8yMlhU0g\nd05T2/tPGKTtuby8fOKj3/rWOrWYE7W/u50pw8EAwLXnrmd/NmbrmZL3OYBC3qdiW7SMgpwWSdTP\nlFo1r6/nAAQ+B+l7PMjBQTNTZmZmuGc+r6NxH1stURJmiq1b/pnj371uj98NuJxKpVKmhCG/G+mX\nUi6VdXyeqDFTmjzs13/zn/EqFm/yKz5/U/WZWqYcDbkUu+19jU2/C93XQMMtB0UAJT/QuHUr7dZp\nQ5zEJ7Yk2jI+rK7R83xMWwC5nP3e7acan9qH3yoWi5lS8Io6dRFArsDJ6e1/lCk//8u/duIgHpw4\nceLknEhwelNoL9iE7y4kCYAiaBl54IMwCE5aTzJZkAu4aWhvG71bgoQfZU9/7Yic7DVEQ51IT/rE\n19j4XooDPpvDUB9Fno4TA8jJOisV9E7WdXmBvZx1RnDnFPZa4Nsg8KY/033fn7r9zyl/NjMtp7fZ\n2CLycgASe5+mGm0qM0qvXDMzJ779Z7ewTkutwjvlpVxswy63JGEPQKnAo1XL3CHQ4W0BFLVKyrqb\nnoY9jG0fro1C3gMnAACCQGaa7DUvd/Lai4IIsuTQ7Y10IkoGI1Itfk8nyMv6MHttNBzqQjRs2RQ4\n4/4mKQcf+XM8SJ4/t9inheXlZWH1O5mSxl2dmscZptxn5CUABpo3/VwQjghoPC3gfo8/c1vSdiEG\nSjyPSpqEADwzeDVvUaQVaE+AnD0lOD9zc7y0YnlGh+WNSLwUQK7I88adGs4QZ2E5ceLk3Ih7YDlx\n4uTcyBRImApMAcPJLTnhskQoz68Ihcm0Nh+fudwKBZp5UZLXR/7kPmZM5uSn9/QYzXk0OFOPBnM/\noWm7vUcbtRPyW53OSJcUA6iVBAQ0ttkKHZ/lEi8w8eRYFXTyZfDndSFhMh3sGAj6DDT0Z5A/zdHG\niMx2Hhvi9kl2IQLmI15yYOAh1i3LnT5jcmrLnyCfMewgx1Mbviv4PH7eiwEUPYF32y5/+bBPx7bv\n876XAt7E0VDQxuB8xC1pLgAQC+EW8vyKIUEIN1l4IZY7otfjGfd3dzNlZZFAJvPH+wWuDF/HtwnM\n6+0fCC0OFWewCMBoZD+uY+Kl3B5rbLGCIXGO11Wa4akXnl3htw4PMqXWI0gMB/zNxrUSgGS2kf05\nI9xtJ/IshjYMNQk8Y6nE2R5PmO5vtvZsBdpBIl1XYmtHS7EQcMmVy4o8wCA5pz1BDCAx++lsx4iz\nsJw4cXJuxD2wnDhxcm5kCiQMElqVlrvhJSNMmO4TURyFbE6F0iJDUhZMUSrK6sqtTGm39gDs7dMI\nzwcFnU4RQCVM9VHJlA/XaaijtMB9PH4U1ggbm+0mgI2nrezPWpEHSba55fIqT7RQM6BhKWC8xoIs\n2/hUOgn3lNX650y/+nMhSp05NnyqZLQoSQCMBLc/vc8kspVV5kkZrl+aJ+QpKUaT/ORD+oxJKCgR\nL4nkSRAQyHsJgLz+9GKuhEJekMSPtWcoRXczJziv5RoNFC70qwAGusCKXAG+BQ4Nt+hKuwPCrjff\nfCtTRkKjc/XXeNiiB0DYDjmD4VohniGd1KLbgqsWxk2mQ8IICpyBizMRQB4qwutLqSrmV6/olr31\no0wJd4kNL7x4C0Bulz+KYY4xx5ou4KjPCGNJwy6mPJq3oLikooQWPB1WSgCCkeDwSEercraLh4eZ\nElx6IVN6jVkOUpg91o0oJbzYXJoC8GLFauMzDSlnYTlx4uTciHtgOXHi5NzIFEhoSCMXNKh4OQhl\nAPCEm0IZ+YUCbdR4nD92MgOzoCDKF//SX86UN7/3fQCbB3vZn10BwCiiRfpwYydT7j95kimluQuZ\ncnH1CrcUWM8RClEWassAogFt472dzUypNOYzZaPzNFMGuqKVGo3hiooD4pCI4Kwn+uko4f+l0pzP\nxIwKbuZVGqW80H5nCKB1SLP/6R6rUsozhAYLKk+xWhMLmVmxztTxjc/6p5OCHAipLi1vZSLxEICv\nMF8uJrjLK1Y7MhAhqOvXDUQo6Vc1NElkML4IoNNuZX/VKoRFnmbSKmYCBYNbCg4221TKSrkMheHC\nUQIgKNh9VxQv5kgi/Rysdq0gV0OqlZbE0z0M4x+dRfFSqyRT0qeAWU7YbZBTjVGscrclQv7eUQhg\ndP8Tjk2ek0QVQd1AF6axFSKVFj1WmnGo+i0Fjge1EgB/wD8DXjGGFzik/pbqn3JL/O7sIi9EJxp5\nFlflVSdpAsCX9yDwzlzzzsJy4sTJuZEpFtbQ42P7sMenchwNAczV+EiuqyInkGfd/KnjysrkZLpH\nr8uckW/+/n/KlKcHAwBPO9zh4SZ3WN98zOOXaGpFPs2oWp1P66BS0z58LZT08C15FQB7MpHWLl7O\nlIFsrvv3aWE1W8rlWePRri5TyQd6t5zhJbXKjDT5CQyOdDxBx7aPE1tOWVixJtXKuX29aa1yYne/\nnSntLq+oP4wBdHsqciryVnb7vFO1iswNjaQwHsyfcBU/kS1ZzFliEWcyr/rYLJdqnEiV6HbkVKPj\nncxj8nOqEZE5ZlNppfgxRgA6R5yTR5axJaPJjKNLdU6LZV298+57mfL5O5/LlMSSwuIQQCm1dELO\nZL8nnKE1E42UPhbw+CNVyA+HPUyTWJZXovy41IwJ/cpCy9XSiWaPNBvLzMwqLz/LMaSH2Zi4w+Iq\nR5tXPfM2K6ihipyuwmLpCiNaeVXRDQSYqjNVAOERr2KoyQnK8pdrBQYLtPVyeZmiKU3FGS0fX4Zb\nlMsDyHlKEsSZdW/OwnLixMm5EffAcuLEybmRKZBwt0d7rBk1MuXb/+vbAG7foGPvl+4Ql81ZsbWl\nooiSwVNFjhU9yLeLBw+ZE9TsFQGkFfrCvZqyP+aPMqWkDI5QaTKh1dnMced6jaN6urWVKe1WE0Bd\nxnBJPtdHAoD5Ou3nna2HmVJ7yjNeqKt8x7MIgDE6HJNuz6gsZGPLtDYuMF88AaYYbZBhQy859s6w\nIiEDZh0hGvO+l+XKHageYkuQcOeASsbTMBLe6x0RDu/I+77xhNP1wo1rmfLclYscrdKIxv5+o9PK\nTfw7Ubrhne2I9+U4TwSUPDkQ+odtABBKSkUJ4Iu2oWCEazaBI4YRYsNWsT4ae/dDAN0uE4KePuWe\n1XpNJxI21EyGHe5TUvhot9XKlLfeJ0isFn0A169xugJB0WGPi6csFpBkyLURKw4QG9YZtDFVLKXO\nuKjGmYz6SLAxL5RdvPcpj/rGdzIlev2L+lYRQJoSkxYEHgfglda2eIG+mCSSqiqWUsVwRvzWzEKD\np36yDwAdLqf8Ct1HeEyAGWiSB7ucN1/em+QmM7MG4njwcubvzwEItFzTs6M+zsJy4sTJuRH3wHLi\nxMm5kWmlOY2rmdLb4+NsVFgC0BRU7IUEWfWCUmDGoTRDQ7RFBxFB1q5M+N22Ig6NBQBzy4zidRJa\ny4sqxLEIYKh8j4Gq0vuKAT2riEZPGHAnHAAIlJbVaurEGmRfBq1f4Imethmg3GoTvzy7KGx7hnXa\n6jNKVauI1zAwFGxsENrbwiKGBMdEesffGaeyura3mIM2P0/sXC7x0oYDXlqlyC2rS0TrGd9xt8fL\nqcoIDwdia9OFdcQ2F42LjRRaGqeA2UeTVzNJDoGzpGRsedrJIGExjQHUFGadNW48pY8VhY9KBo+E\nxD1d+5hmNxYXdjsEMFPl9jnN24MNUjPff0zlk3t/mCmtvVamdAYcWy98P1MCKLuq1wZw5yYpjP/G\nX/tapjyjFTgscbSDLscfdnmieqqkpP4RpkneV1mMJsHChYkcL8ZAWTvg8aMN5hjWC/ylHG3yjGFp\nFkAqwsvcFlMaq89wuYZ14S9wkZQ7Sh9rcZADVU1Fe3QgFAYhgKhNuF1sMnw/6guPlwmZWw8Y6y+U\nCQlnLjCC6SsXLFXi1RApgEgLL0zOxITOwnLixMm5EffAcuLEybmRKZDw1ksMNGz8gHn9tdklAK9/\nidsr3nqmhIqPGBrKiV8tRiNTZoT43nmPEY1agyb0M1fuAEhl0hcMYA5YrNNTZYBnTTs04A/fezdT\nZkvcUlGQsVapAdjcpm1sBRueQOK80gVbB7R4D5pU7m/R1l1bZqJdoFGdkKDOq4iF6UaKkEJBHFMs\n9GPVIYaP0uMppOPooRSrIzEOAMO2DdXZjFQ9DyGLSm0GE5Aw5xv/gTqXlHXLrE+M4rjjGM2pwWSQ\nP3/y88/ChI/X1zVIzuRRm8smHg0BPFHd1YHoIrod4v3lBaK5WpUowlfScmiUhAXx8+n+dgc9AAMb\ntLjkH21yXT3YYKi0F4quQ4FjVHh8IxW3aq3tR58A2BTm+qPvfDdTbt98LlOWGsRH/U6LI1F7m9Ft\nMpR0RLl3QorCdKnuIIw0RZDZk9JRlVvn1c9nSj34KV7REed25Gcs6UZEqQhjmSfqxqK7kCtgJI6E\nvFZyXwz6lsfZj2MAvQ7PUtXRBtqzqJ/h/AxZQGI9HDpaclDyankk/r9cbuJCMTp7OTkLy4kTJ+dG\n3APLiRMn50amQMLKLMHOs9cYEMmKpZ69yoaXiyNihtZ9Jl6OBFLiiGjr9V/4m5ly+dqrmXL1xfVM\nefNtorm52iqAzR0a6oHYvEoKaakmHB3l9R0e0MaeU9dMsx2tFnxxcRHAQFX2e8oAtFKyWk1RSGWH\nhgo53X+8kSnLcwQaNy4qNe64fP1f/Rse1hJHZfrOqEfm9auEw6+99ILOyK9bcmkWiUsNv8g+jzSl\nFuQqFIUajABDWY4Lc8pZtQ5shQIm2AKQl+muoraWAqMtMa4dHbYyZWQ5sQrwLShv8Mb1awDyVqFm\n3TknQOMJ+c73fpApRtVvRZG9QQfA+vYT7UCxWZpT5nBVgdGizpNXKmmgvEdPbb56gxBAoLasqeDw\nVlPE5wrfVmoNnVMEJB2r9eOZrAS1XqsD+OlXX8r+7B42tQNx96NHnNJ79+7xIwXPH+5zSvsKIJ6Q\napXrLdKVjmK7C0RzRpeSEwour3B+2urqunvI0eZ8H0CoZmUFC8C1uGck5F9Ujndba7JkHQ2MLlE+\njWFWnaq2DId9zZvwa0V1jjMXL2WKbx6GcWc53eBxPnIKjNdTcnbmqLOwnDhxcm5kioXlF+k2e7L9\nYaZ84dXXAFRnaZj4R8biIONCr9wfb9Ab93NzTOZChQUfM1VVPwQ8frlQwUQxhPmS19bo8P7wHot4\nCnJJtuVTvHqJ1t/N51lV32yqg0g9B2Bzm4knnt4SDfFhtcSUZDZXudLIlL7K0D99pPKgwvRn+kD+\n7LCv8nSZM51DXbq2xLef57dSI/YVL22hjAlTZUx2LFNrdp4pPGMiB+t3YvwNskmtACrhvzzausqh\nnuxwWpr7tFX7fdWRDPW2FKODUQtcvESf9OVLFwFUC7ZsLHRwpoX1zl2euqISDbvRg6gHoDHP3DG7\ny6GMmp2O5laXXCtx7UUyFT1rrarWSl5QBVDoqhvoiC78ZrN5Yth2a0MRRbR1RqsGu7zIZTM/fwET\nFT/7B5zJhQbP++rnuRQ3NmmntwecqI9UuXKaTJwXKD96eYYX2FHKYaBVGltClipaPC2nRMliOV+x\nCM/HhLN8JPKSstomGbwwW9V87bHm1nrwRCqJy5dzABKlvBnJnfE65CM1NrZMQ323FNsql+VpnNXI\nYeJ25M5eTs7CcuLEybkR98By4sTJuZEpkDBfordyMDBoMAKQVyFLpWquUDr/iqJbnQloQ/7WP/8X\nmfKrf/ef8LAqUygUzYaPAFy99kz2506TzteBHJ+rywQLRlw7VCuUa9cZAXjuOrHh4dvsd9I96gBo\ni/Q2iszFSyO/oXyZpEWre3ZO3Axy1fseL2RjcwfT5O/8rV/jkOSirp7qE1kWdDLO4XZbbAoigcgH\nJQCB8llS2ed9ZS2liXLQhCby8u4HZsznrdDnGKK0fJaBaA+MsWCu0ciUWCyAJZ/jb+0T9Ww8Wc+U\n64q3+F6ACdzqC6V+RmnOkSXCmatb2LDilQBcvMQ8pnDIkext8yt7TQZkVpfJBldapCu32drXUblz\nfY64tVScAzAQy0Yv4pyXKrrvEe/7uLerHPbm3Ih6HO3rP3UnU24+uwZgENJr/uDH/Mq9Tz7IlJ95\n7cVMuXSZS/rRe4xKmb88OaPopKBsr4LyChPR3ZUVMInEgHjUVutTEYSUZolbV6qKEaUJjrUsFQOi\nbBRf3oNxZOaUpCoPMkgY+ykmGBA9KQVDnzrsUOSLRtMS6NpjTfu49VQSYKJwzSgqT4uzsJw4cXJu\nxD2wnDhxcm5kijWYU3FAT9Bs0OsDyKu95dG+akRUiJMHQcSFBi3DTz9kKsrmBhX0iPgePl7PlC+s\nfhHAM88yJri2Q6V7jzvMFxqZUm8QG96//4AnWqPV3RLIGsl8fbqzDyAxqgRZvD3F9bxTDAxW1mMB\nrIJ4zsL9bUyTRMloYxtbH9UKrJgplzhjfdG29UacuvX7vMZCoQzg8lUWsj94zPr73/uv38iUSNGc\nko5WMUVAslEn2GnMEhF84QsvAVhaZHnEcxc5XV5OnIKy1C0SZGGj/jLxxdqFBpVn2Kwo45DrKbtn\njILPfvHlRcy/uMwxWOB1b+8xgE5XBAYqzbBksYaYyNeu3ciU+iyvqL5IkLin6HAi7JxVofTUKLSn\ncFsYKrNJJAQFY3kMeMsKYmpfXuWULs1RKeU9AEsCnnWlL+0/JO57+OP1TFlV3LO1/X0edp6jDc/A\nX4F4C3w1iC3pZ9jaYXCz2SFlwu4Wo5BzM0yZvPMC0ai1K874D0aKx1lU2parNSUwV0NuDPC5czwO\nR1o8L/vIvqtqm/F31VBHZ7QlZzvnlRmXt2BgCgCeEG58dlqfs7CcOHFybsQ9sJw4cXJuZJqBarUm\nCg1cWFzABBL55rtEeXMKAN2Yp7FXKljYhfhrd4cgLhm2MuXydVJ8+aUigEqdRv7iClNM91VC0VJw\n0PgBl5eZRRkIn1oJjpXvZ9FAK22xDMOBIoyRWMYXBSvgqQmryMxKinFEYhM8Ib/7e3/AsYn32lPy\nXk3h1BkhtSs3eGlLC8RHCxdYtTO/uAygJDaC1kfEF+9//ChT+sKvBibsvsyIrv765SuZ8qUvvsLj\nV2cAVH3V0MjEDjVdkdpk9awiRw1ByzpsoyG+/G02RtvbawIoq45kZZUTWKko+/eUNATnJwqhRMIH\nD0BT5HnttlIljfNbKO/RBgdQb/O79dkGdxYdXE/5rshFmKwvqfB2lCpWxGMAhzNZ0z6B2pdeWuC1\nG1tDt90CEAlgGp/9VcHVjz76cabcvPW8js/Z3lQqaUmFVifE4Jh1BkiE1I6ULL27S+/EQZNH++S9\nH2bKx+8Se16/ziKwK9dvA5hbFAuFQJaxSxpPv6Ev3+hGtC0Y9yI41mtuoh2sgo/a08LFpzsNm4yD\nj2POkuws9lOd3lsPzsJy4sTJOZJpeVh6WNZretPOlDHRUrQtsqC9Fp93i3UepypPZKQOKOvK5VmZ\nb2TKs3oJZJkyP3zzo+zPJ1v0ns7UaHPllYHywaePNDorPVG6hx7GnS7fvXML8wAi7bCtGp1anQMI\n5HSv6L1qRSEImfiT9DiY1eXpxc8/evt/Z0pZNEzDkJ71vJzKP/3Tr2fKwyeki92n2xR3PscyjkK5\nBKA3pHWQlxn7yiukOhqoGap5iG9cY9nT58SytLbIS6tXaPskgxDA4232B905EAf0Hrd0O/RJt1Qc\nHo7UKV4nsnJrq8EajSIAlQbn5A54FbOz02cJQHC8JhkTL8msXNnCI4FqtmxLocTDLi7R61+r8QJL\nCjgEGmSQ543IctBSFYJY36NZ5aB51u1JnFCB1bgMlZqnMus04rTE8RBAqNKTvi6nMsO0xIfb9I5/\ncJ/Wt9X3jFT2lLbPTHrKxEyVkvjBn5e9dv02oxa9I973D95i7uFbb9DC+s531jPlww/fB3Dr9svZ\nnzdu3c6UxlwjU2w5+f5Jw0qVXZNbtACSGBNZhCZWrBPLmE/GKWBnypgVLudjooouSs7M63MWlhMn\nTs6NuAeWEydOzo1MY2uQg+3C8gXt5AFIlLBz4SIhyY821zOlBTVrCegmbyzSLTdbp6Fu+ThXBAlr\nswsA/uVv/uvsz56O3+6z6qKn8hrzN682RJXV5Km7RTsRvaQffbwJYEe0BObKbXgcZL0haCAWpCCk\nMR/0mAY1XxGOKE03aXcfE6XOy8a+eJEe6Bc+z2qhvGDF++/8MccvO78mkqOdvS0A1TphxUKdO/z1\nr/08B6kcp9lZ7rO4wOybZpMT9eAh6acPW4Sl7cMjAEeKWhyIhqmpfieRQhAFebgLgvNGYlGvc/xW\nxzO3PAOgaFC6LGoBUVaclnmRTSehebh5oiTqAyiIZWF5ZS1Tcqo9KiiryMBpSZUrvgZptBbG/pzl\nBFmiWa+rQhxjgJI/PhU27B1yJp+scyabyhFqqKvrykIDQEl0EeYYTgOi+EClP3tqZnNxjUuupmtv\nD6a7k61kx1oRp55tkWNbmVmNBdYn/dxXuOSuX+dP8rvf+lam3F/fANB7WywUYih58SW6Gi5d4kEC\nRWbiyBi9rZBI12jO9DTFRD9gIxCx5k/GdTXuA2ttay29y+qTxk53D0CSnsSVp8VZWE6cODk34h5Y\nTpw4OTcyBRIa8W59jsZ8FAcAijJ9b4r590dvEFsdFljNn4Dm98pFHvmDDxm/+Nmv/MNM+b44c7vd\nNoCRAnM7WydDgZ1I8SNht4ZH7PZMmdjncJc2fOQ3MmV1pYEJa9YqcgbiQe70xMWsjqrRgGVDywFD\njWuiUR5GVs9xTJ7cZY1+W7GnX/3lf5wpX/vaVzPlD7/JaNGy+CGW1XW1rFSgUi4BsCI+3xkpJSVD\nRbLGDRZFSmPZ/oTDfrTDNKVQ7XOCUhXAzAyzfpYFZEbhyfhOXkjQSuRNmZlhkK5en9FHOQAdEfI+\nfcp7Z3N7WioCSpGnsJqSzhr1ZQDJmAaS96Vc4+lSq+oQbElSbTlFs5uaggRApBsXxRxbe19k3Hbt\ngoSdQwZPN9XCZ3Ve1U5Vpv5lPZwSQdFIh7Fw5DMCWbduMtPw5Reo3L3PMPHb732EaZITEvTEZeyJ\n+CTvW6GMsqIUxfMUGL1xk8TNiX4ym9v/AUBzj+A0UXHY0ycfZ8pzNxg3vP05fnd5RS4g/dKjkfia\nlcwYpzEm7ssUamzh7tMkfGOWx/HF2pdSYIwwxxU/p8RZWE6cODk34h5YTpw4OTcyBRJWa4Qtc4uM\ncWQ97weqXynVZC0rePToEYsGvvw62c4GHSVn1mlsb22wnuDTu3d52CjEuDEHOuJdqM8TilppTkMp\nrLdu8fg/fIeW7Vsfr/PUX/mVTMmIBu/f+/TEQSzXdKDqisurRHMl5VvOzwuMiJIwCqfnsA16jLu9\n+HkWyv/SV38pUxbUKfZnv6hIn6DHjCqK6ppkv1DCRDdQi1sZS7c1CqrLUE9EDHFNs7F8kXHJ5gHn\ncKbRADASWskJLxlvt4WlrOlLR9G0VC1SjFb88RYTXgf9HoCRUHasEo1K9czSHIPktQrn1vDdzu4+\ngLYyVy1f9PotJkYa3bufNzRExXBxqIYtPVHr9Yc9AJHyeD2VHCVD7lkTCjaa/3JBJV+KfzXkE5gV\nyXo4HALoaZBGN+ipoGROcL4iisqNxyy0EqrD556/gWlihP3+WJErwOqIrHQmORZcAxAK6V+8dCVT\nrly5CuANaycszsKdnRYVocWPPmIXq6tXObbnnqOyssJU1RklxyKXBzBQW9ZYv4684LyFAi1x1Cpz\nUuOxHIutzxwmi4Qcp7sTJ07+Aoh7YDlx4uTcyBRImETEULPzREzdfgygJ3xhUaTLl0lCcPd9orzD\nnpIDq4wkXiZhN9Y/Wc+UzU3ii5/50msAusId9TXmDc6vMbbyqEnc11NL1UKVNvzsMo//Sp1j2N1j\nDGh9fQNAR7X7LbWWXF6i2V8HjeErNWK35brI0UEcESrGVD2DS+za8y9nyq//g3/EQcYEGp/cY8wu\nyYnEQpHEkTLimi3Vuyc9ALG6ZipGhATEL0dtFuv7T2n2byondihUkigdsaoo5P1PNwA8eKTAq1Ix\nFxYX9F2l6aqR6p4mEEogHDMdSqlVygAaJZ7FOAX7nemxVEzkozb3OOz7YmqPkiGARoOlo2trpBYI\nVao2Cgknk5RDaguJ9/rG5DHUaIWh8h4mcF9J3BJl5YuaTyBRuK0qBkdDZAVV2Nlqz8KpRi6Y80/G\n7Eai4d/YZ+Vmr9vKFCuoXL1wEdPEF1wyBToRcgrsjtMsT9X66SOrQKzP1IHJuk0pRrGvyHu7yfvy\n9h7x4wfv/ihT5lX/u7rKn9vq2hUApZLynBcYWFxaoRvH0nftlkXyMFjr1nHiqOWdJh4mWBzSM5jv\n4SwsJ06cnCOZYmEdiVKgLA/xcBBCnS0wkZi/NM/X9V15zneafAHu6Z3cmOGj9/aLdEnef8iclIzA\nypziN2/Qc3zjKq2y9c1WpmSl5wD294xfQd1f9G7ceJ/m2NZ+G0BOIQJfFf+rF2m4XdFT+rJ8+cZ+\nNZQplyR5DXJ6LcWv/f2/xwGs8p357vuMKpgHNBy3CVG9hVy25lbM+prE9m6xHp/jVwm3hHo37u3R\ngrNUI7OEGmKJylzRzX01Rpe/dm9PjULFFxzJ6W7FOtY5pqK26SUlH3mRDyC0jjRqf1JWatVpaal+\naGuThm1VlT3Pv/AigAWxklUU+hiI3fjggGl3xiTRE61CRXlqs3Wu0qp61pcLeQCBbKVYTvcsyANg\nJKLqgXV2GXP+iqVXeEKZbQj8AoBULVcHQyr7uzQY9/YZX7JqMGPCMF6QokiNT0guNQuLW8xFnZOp\nYtwGExUxVMzn3e/QHt/e2gKwtUWjqX2oCjkZhjMK+9RklJXEO2IUchvbXNJ319kNdzCIAUQxD7K4\nRFR05w7r7W7eYDLa0hJva32WkZNimU+AFFot+oHQpjfabud0d+LEyV8AcQ8sJ06cnBuZAgnv36P5\nd1nJ+yUvBJAIRARmQ5qHT07lmkiBn3+eqTR/+Af/JVN6LVqnlQX6Vu9t7AC4eJH+vKu3SO9bFCR5\n7ln2kmk1W5ny4Yf07icCIxsHtPPbfdn5cRFAu0WkubxKG/VRk1vmLzYyZV+QB+qV0pK/OVVDoEEy\nxDR55503MuW9997JFE+GrifT2koczOcKGCMCjeqg4GFiJgtjogLx+SpFy0v5Ub1IL7UnAozIt2tX\n+lgKAAUhkUgsgL2WRRV0XVasIxQaynsdqW1SRzioUggALIvALxAuK5xZSoH5Zd7ueWEEYwHOFtKR\nCqSOOup4WsxraOLVkxv+mRVGToq6d771jlUxVnfQBzBQsKIlXGmQbTDgGW/fJjdeXhmFE3zBJ1v4\nDLtHADa26dDY2aWvOhSUNnIRyywryFXS0TV+4xvfwFRRMldiOVaR6mOEFq0PVM5X0pMglS83/Ltv\nvckztnYALCiJ7PEWr72uZLG8VrgF2epKPQvEjlIITnpgOl4HwH6LgZr1ByxQax1wWt56QwtYpJiX\nL9MVsyZa8Atr/EmurXBLtTYHIFcW5YN3Zlqfs7CcOHFybsQ9sJw4cXJuZAokfOdTBqEu3yEleYIu\ngJzFy2S1ttXPo9VioGRh/uVM+ZWv/WKmvPx5Wt2//R9/J1NyKvWenZ0D8Mwao2zGue5HDBLNr3J4\na9eICFrCIG+/806mbHXEvZ2nrTu7ughg8Tr/9AXHYpnUd9UI59620rv03LY6la6uNUpsir6FCfmj\nb/+PTOmJGq2Q52HLqkGx6fVTVfZbG8u8QcIcgFLxJMouiF8hUOpZSW1lDWgodgevZC8ehV3CEMBA\nZTEjBcgs88heVcF4i65UkHy2RujRqHBLreIDKAT8Sl4pQrl4OnDGRGcUS9oKBHuTDOxYDYoyniz1\nrSTcN+hy/P1DLrm+uq8GBZtSEcXFEYBPPiRaebi+zpEI+BuSWrvANKJ5kSP2BetMOZA7otnaB9AL\nLf/L6EC4xXr62s2oCFttqbZpW7UyJ2QkhG4h5lwk2gZDi9o5VQqVhRQ7Cg4O+jzOrZsvAHjl5dey\nP994jy0I/vhHzLE6FKl/rLWxvMqQ35e//OVMCXTL1tUs9vs/+D6Az71ALv+6XEA7uq5tJQlaTHZV\nJBBXr17hGRUT7x4d6opSAHm1sx2c4hQxcRaWEydOzo24B5YTJ07OjUyBhHfbBCN7sagL8gMAXij7\nLRH/lrLs1tRQ88tfYqSvlGfc6uqzLPj+q3/71zPl3//Of86U3e1DAJuHRhvA/qwFWbzNHpV7YoOA\nIjLpIpHm3DJHa2AnSxlNSrZdJGRCsoeRijbGiZG0rbsezfuR4l5pMt06XVmiMbzVZ/wljluZUlez\nzEClOe091moctWmHj2KLfw0xtRZBHGaFMufWMG805njj+6ai1q1VkazHWVauHVb8ATkBqJJwX1mT\nMK8U3EtSLq4xJCcgjsHgCICndrOBMEmjXj45fsndTz7MlBfuEEfYtGej8xSaS1TDYbCiJwb6QY8R\nagsXWqPca6IzX15mgmJW+REoVjsr9kQ7ryXlWvLnRx9/kilGUGHOAcuizBZYR0mhfXEWGiQM1avN\nOOMfPeXasAzS+IwGVuO2o2P2dP5vJHlCzEgEEi2oWVY4+Mtf+ao+8TDB137zZbp3XvwpKgqujuff\negVcu8bM7UAzduUGSf7WLt8CUC7zdlufARu/9Rkw3Le8xNRxo3zwhZQ9eWniZAhgpCtNctNnCc7C\ncuLEyTmSKRbWJ2qP+rvfpaPuC88uAlgtqHm3XiAXVvnsvLDIl9hz11TbqRKKrV0+cb/+b2lYvfU2\nX7lZxc9E6YucpnLXxSUeNjY3s/zl1ic18tSI/PilDELrPmJ9t2kn+LI7UtUMR7LO8lY6Y0lJ4fQq\ngXSkEvEq30JH1jUz5kv4+du0KZILtLl29zgbO6Lr7bRiTLylYyVSJRGPVg34Xnr+JfJQb8q5uyt/\nfz88+drPSn+KKq6q6pY1qpyuJTX7WV3jTbyu2uOVEqeuo8SofdXHZh7uquIAtRm+aRcW5nCGjJT0\nNOhwtJ6sy8yKMHos63h67y7tnSMLaOidnFdQwhrfJ1aqbWW9cQpgcYGDNHuqN7aeqDx69PjEPqZY\np/ieCrAPWy0A3T21yw1s2Lwco+jqKk0pUo1RPO7tPt126PdpQvpKHwtEBh3qpxQp9zDSldphjd3M\nqneiOMJEM5tQ1uva5au6QhWHSfFEmvbgETPX+qGhFrFmz16dPN3BofpOaTaq9Su6UNX5H/LSNp82\nNVqOsqj6uayyKFdTdfrBmU2YnIXlxImTcyPugeXEiZNzI1MgYUd22jfeYh3Mp/fuA/grr7Ig+7k1\ngpQH90lD/POvkau3lKer+EiI7Lf/G/M+3v6Axfq9SAUxQQmAJzfwuJeknO6pZz45fjQUZBtpS07J\nNUMdNjM3g+AkuKtUZH/KtI4NQ2ge7ESR2mQWlB12QvY3Wcgej2i+9mXt96zHqjpfLolAKj8kZCuL\nYKHvpwDS1ICxsIP8jr0+weOXXyfAvHObpMyPHjE7Zu+ATn0rE8kc2oabSjrdkiBVQ8xZsc64vcej\nfSJeJMjnWl8mvKrM1gFUZvjdebFr1eR8PS1l3YhQiMxCHFnQxpMzuSDcak16LDJQk1PZU2ZQRRcS\nKWfn7sek62g39wG0VA2TWLtcHS3QkiiJ5MD4LnrC9bsi7TJI6HsBgDmth1B79pQSFokEIhkDwJO0\nCrncdBPh29/+nxx8RMLiipKSElE/G/nHGIQmlhopwmgLESQRAE9IbSBwl4z5sKz+RlGXBmMstZqu\nUT147Dv00Ht2B5UEZwmGenpY0MMbE4+cuvZxUmAMAFUdRIGs0+IsLCdOnJwbcQ8sJ06cnBuZAgkX\nFmkZNptEJVutAwDfU6OaePSs9qXVtyQSO/i02H/4Bin3fv+b38uUYVLVOcVq4B17XMaWYyVD0Zqh\nGieslddYjCZnBSXGkeD5mMj1mDH227H5OjpxtEQkCmZar64S49TrVN7AMVlV4G/jEbFhNLTsGCoP\nFO06VJ6UXXBX6V3daAQgiQ0SiodaIGI4IOJ467v/PVN+scoruqMr6os+wUJmWR3VwCJcAhE7exzt\nQxEW7/UY9hoIHpUEAOeVXlea5fj9cgEChgCKwpU5f8pC4iVrkAZGPNVmZaMd6AItx6pkeTpSLMAX\nNulYeGw0x8ZZbKHeoICJkqygZEfjkELh/Y4oPSxuaB1hLTZc0vhH/RDASIwCFpC1AJ/5NCxzKlKi\nYhob7J0edC5phURCgoFqwoLCrEaiuKQ5T8b83epVYyCRa83CiOHx7ZOXaI2UbA9ROap3VDhQ6dXx\n32ykJrgGzC030NNoPZzEjyZhx259BGCgz0vBHs4QZ2E5ceLk3Ih7YDlx4uTcyBRLPpAdmy+IQmxQ\nBPBgx+okmPn5C6+Qpa/cYEF2W5zov/HHhFAD2agjGfxFMXtlJrTlTJr4MiZPW8/F00jQdlYtTrlU\nxkQmm5GyH6nhipVHDAVSZhus6li9QKUmHNETI8UJuXyTJGTtLiFVd8PsWHG/Ceg1daKCqmpChQXj\njLHbYLAdIrW4Erfce4/x1sdHnMklT+1XlS4Yy+rueAmA7ZRo5Z6ikxtiBegZAcNl1uivXiGbWkm1\nLDCgJ8q9Wq0GoKIonqfE1PSM4BeAtpg8ekoc3dkUB8MgBBArRdYoJUaCbHZd1kI0L6KIcRRYit3x\nbMKMm2HQURxZRAtHh1QsNludUVKxJjAdKTAtFsMsr/VQ3YYMCcbKyTRi+OTU3TSCitwYsh0TyxPu\ndBjwrcjFYceyTsAWCgwjG5syLT2LG0YAQmPpEPeDJZ2Ow4WG2cdI08alORT+zb5lZXATFWXxCcUa\nqXqnfsf2USAgORpFAHpzXFcXLs7gDHEWlhMnTs6NuAeWEydOzo1MbaRqPT5lKwYlAGFEu3ynQ/vz\nzY8ZsvmVHm28o5QAarNJpaQgXdSzHDYRhFcqAAIZq1Yfn9OoLKxgMcFUHAbGhGfFZZ2Qww6jLgQM\nMVH+bgCwO6ChWxMSnFtmPZ2RiH/8gCHRfGK27jGpNxhKW1phKG1LkNAsYKvMH8pOtp5RsXo3xTgJ\nH04M2w43Egbp7jGtzis2MsUX68CmTvQOhgDuCUB1apy32kUW/S2qbe2irr2o5MwxF5+gTVEM9H7g\nA/CtyaiF87TltGw//FQHO1kBl0XTAjG4Ww/OnHUzzRMWVUSOmDuFXyIVhHYiRRKHEYBElXFjAjz1\n+yoUGYlbfoaT0BVcbR9QsaZn6TgKmcMEgZ+xOKTpyTtl2DBvRAu6y73edA/DxiPSDX66zfNWldQa\nCEVG45XFGYv1UaKgc36chj2CKgoB6NLHLgZrEGtd+8YxR9tH99dmO2OkSOKT8VBPvo6cGErG5PRa\nRafmCR3l9MZzFQDPvMQmEnUlFJwWZ2E5ceLk3Mi09JkxZY96cnh5AIn1mJSZs77D18XXf5utcb76\nlVcz5f4mrYDeONdJvnxVV/iFAoCK3pkF2Up9FVWYvzyVcZQv8dS+3vnmoLUt2aO9b3k6uhzboSHj\naGGVsYLdPdaRt1SV0npEu+D6NVW3H5eSOtYUdTn2covlr5XfHFHu5JSOi/azneztY/vpLZdK6egt\n95Fe8rNqqPPxgKzW74tdulmvAJi/xMGvXSErWWON116ocPzWhHWksQWya3wpgd722Rt1bCLl7AV7\n5pvPT5SmFJu7VzZLdjRL2EntLc3vDsW8HIkbI9GcTvAfUMzpnnUVNesgkKkVaxWViqpYUu3RwR7t\nmq5iLHmtdt+6ew6HmOhhYybweBK0kq3jaUlLriPaiV73ENPEg1aRLYTYiKRlAdkk+5pJGVDWHtWK\nsbI5tilNlftmk5sadDCCCuvBI/7uWJ+NzJTz8wBS61Rk5F1mnVnb1/H8KI6h8EgkMuu6mEIuvngT\nQJDj7WjdfR9niLOwnDhxcm7EPbCcOHFybmQKJJxXU0mrmehGIYCCOi9aKoenRK3v/PC9TFnfpBv+\nsEcbuykPvTJCUBUYySoMijqI4Y6S/OW+zHL7yGzUSEAvZ749mbhxOMJEBkpZSHNxntQC80tEgqHA\nwlB1/H1j7xWgyLpynhYrlO+qWH+mwRMNugQyxv0QyyqOxwa/xm8W9HFJhX1SJUN1lWLzXXFVPxKF\n9H5FuUgrzA5bvbgE4NoiowoLCi94mvyuAKDVQwTywtaMOVo7B0qdK5UrAIqa0rzIOT5DzNU9ZgFW\n+lOa5ACkikSMkaa+ay722Nz8QqnFohwLWiTG+pDy4OZv1u1Q1CJU+lhP6UXd4zUiAHIFHnagPMFs\n/FoyY0xvkNC2GBtEGvLUB/vE7KPwjOVkpJXaYSSsbh9BxTqWg5gIf3maWyPqS9IIkzBcnpmCrt3w\nZZIeQ+iYyMMajeSrNy97mmKiFe6YhcI8C6nc/7mTP9WRqC7nbpKC+eJVlvQNnu4A+LH4NsoipDwt\nzsJy4sTJuRH3wHLixMm5kSmQcCgQpE4rGCYjAHlxM0RmFRv/gUJm65vkALA6+yg0y9a646hZaa+D\nidiKlexUraFLhdjQk8FZMP42oRWrvN9tijEaESZKN+aU1LG60KCyykhZS9it3WI9REfdTRrqfLO3\nM71wPFQxhF+gxTu3xBON1H80UrhwZJE4494WJMyuzDJ3cqeCg1BVRyDeu1FZpS2zHORzsysaNqtq\navUAwIygYlGVRgOr6rB4pdHaiT9vjBY0hrwgeRZpzWtPS8hKz6AqBzAIrfTfIlbH0nw8XaCRu9uS\nmIB7wiCWPWSwKzm5wLJamZHSCX2t55FwX6zDVrUUDQl6RpLRV7HL8T43yal4riVkBQLINi3Np/w5\njIaM3uZOQn+JBQBF5+ApXphXTA3x+IfHnRV5H5M2mIshzQEoCdg26mK4twvRsM25YT+Zgm732Pmj\n72WRRPtK58hi8Tqs7mbbgs6LPPXlmzczZX6eDoqNj9goa//efUxknJUKZ02Ts7CcOHFyfsQ9sJw4\ncXJuZCokpDFcFAbJCvsTxS9yFqSwou2xchIJpslJ0z0dl3onmLD/Dw6I6ZpiSa/XWJAxK4BW92hM\nJurtGSVDXYkgQMkHMBTlmDGIB7KWo566MPW4T6fF7luJyAyM7XsQTH+mB0oTbSwQnNbESB0PxWom\nKGgNoNIxmZlRC3iYQCLWm9aI0AJBzrKyEOs1VZbUSO1WE1NFTbA6s+pDcfJ1NNqeAQHjNRcrQEGI\nzACgAbEx/kpTAKGK7AsFKUo1PC35ovE1KnPYPAmehwmmh3FwcJxmezKwCEUSLQJrlWSRQloZsX1f\nSDDuq5hGUcKqvlKeZeDY+OdGKtvyToE3onWL/I4bnlKrCq522/QwtJUvaojZ7jtUlcLt5mcx8nWV\nSKXim/RVkWOKDXLM22d1NrkUE5yIvaCtAYxBob4rX42mJQjtbnqnvnVMIo3Nzmvp5fVlFYHduqZj\n8UQf//AHmTLc4e/Oj2NMVAslZ7SbhbOwnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJ\nEyf/z8r/Ab+8NWulkQjIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7F632C0DB110>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next() # 一个batch返回4张图片\n",
    "print('实际的label: ', ' '.join(\\\n",
    "            '%08s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid(images / 2 - 0.5)).resize((400,100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着计算网络预测的label："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "预测结果:    dog  ship plane plane\n"
     ]
    }
   ],
   "source": [
    "# 计算图片在每个类别上的分数\n",
    "outputs = net(images)\n",
    "# 得分最高的那个类\n",
    "_, predicted = t.max(outputs.data, 1)\n",
    "\n",
    "print('预测结果: ', ' '.join('%5s'\\\n",
    "            % classes[predicted[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经可以看出效果，准确率50%，但这只是一部分的图片，再来看看在整个测试集上的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000张测试集中的准确率为: 53 %\n"
     ]
    }
   ],
   "source": [
    "correct = 0 # 预测正确的图片数\n",
    "total = 0 # 总共的图片数\n",
    "\n",
    "\n",
    "# 由于测试的时候不需要求导，可以暂时关闭autograd，提高速度，节约内存\n",
    "with t.no_grad():\n",
    "    for data in testloader:\n",
    "        images, labels = data\n",
    "        outputs = net(images)\n",
    "        _, predicted = t.max(outputs, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum()\n",
    "\n",
    "print('10000张测试集中的准确率为: %d %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练的准确率远比随机猜测(准确率10%)好，证明网络确实学到了东西。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  在GPU训练\n",
    "就像之前把Tensor从CPU转到GPU一样，模型也可以类似地从CPU转到GPU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.9488, device='cuda:0', grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = t.device(\"cuda:0\" if t.cuda.is_available() else \"cpu\")\n",
    "\n",
    "net.to(device)\n",
    "images = images.to(device)\n",
    "labels = labels.to(device)\n",
    "output = net(images)\n",
    "loss= criterion(output,labels)\n",
    "\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果发现在GPU上并没有比CPU提速很多，实际上是因为网络比较小，GPU没有完全发挥自己的真正实力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对PyTorch的基础介绍至此结束。总结一下，本节主要包含以下内容。\n",
    "\n",
    "1. Tensor: 类似Numpy数组的数据结构，与Numpy接口类似，可方便地互相转换。\n",
    "2. autograd/: 为tensor提供自动求导功能。\n",
    "3. nn: 专门为神经网络设计的接口，提供了很多有用的功能(神经网络层，损失函数，优化器等)。\n",
    "4. 神经网络训练: 以CIFAR-10分类为例演示了神经网络的训练流程，包括数据加载、网络搭建、训练及测试。\n",
    "\n",
    "通过本节的学习，相信读者可以体会出PyTorch具有接口简单、使用灵活等特点。从下一章开始，本书将深入系统地讲解PyTorch的各部分知识。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}