23 循环神经网络和自然语言处理.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 循环神经网络\n",
    "\n",
    "## RNN的结构\n",
    "\n",
    "左图，一层只有一个神经元：每一轮的输出会成为下一轮的输入。\n",
    "右图是一个神经元随时间展开的状态。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g94b0dksi9j30de05uwfe.jpg)\n",
    "\n",
    "一层只有一个神经元的话，拟合能力不强。\n",
    "所以变成，一层有多个神经元。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g94b0v5wgaj30de053q43.jpg)\n",
    "\n",
    "单个矢量 x<sub>(t)</sub> 输入，时刻t的输出y：\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g94b15ckpyj30c001et8o.jpg)\n",
    "\n",
    "一次只计算单个矢量，速度太慢，所以要变成一次计算一组矢量。一个批次的矢量构成了矩阵 X<sub>(t)</sub> ：\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g94b1kcjldj30de02zwen.jpg)\n",
    "\n",
    "其中，\n",
    "\n",
    "- Y<sub>(t)</sub> 是时刻t的输出，m行，n<sub>neurons</sub>列的矩阵，m是批次的中实例数，n<sub>neurons</sub>是神经元数；\n",
    "- X<sub>(t)</sub> 是时刻t的输入，m行，n<sub>inputs</sub>列的矩阵，n<sub>inputs</sub>是输入的特征数；\n",
    "- W<sub>x</sub> 是当前时刻t的输入连接权重矩阵，n<sub>inputs</sub>行，n<sub>neurons</sub>列；\n",
    "- W<sub>y</sub> 是上一时刻t-1的输出连接权重矩阵，n<sub>neurons</sub>行，n<sub>neurons</sub>列；\n",
    "- b是每个神经元的偏置矢量，大小为n<sub>neurons</sub>；\n",
    "- W是W<sub>x</sub>和W<sub>y</sub>的垂直连接矩阵，形状是 (n<sub>inputs</sub> + n<sub>neurons</sub>) 行，n<sub>neurons</sub>列；\n",
    "- [X<sub>(t)</sub> Y<sub>(t-1)</sub>] 表示矩阵X<sub>(t)</sub>和Y<sub>(t-1)</sub>的水平连接。\n",
    "\n",
    "> **神经元的记忆性**\n",
    "> \n",
    "> 时刻t的输出是时刻t的输入和时刻t-1的输出的函数，后者又是时刻t-1的输入和时刻t-2的输出的函数，以此类推。对于第一个时刻t=0，因为没有上一步输出，对应的矩阵都是0。\n",
    "\n",
    "## 不同类型的RNN\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9dao2wjj30j40cbjut.jpg)\n",
    "\n",
    "左上：序列到序列，sequence-to-sequence，股票预测：输入过去N天的数据，输出过去N-1天加明天的数据；\n",
    "\n",
    "右上：序列到矢量，sequence-to-vector，除了最后一个输出，忽略所有，情感分析：输入一段sequence-to-vector，解码器是\n",
    "左下：矢量到序列，vector-to-sequence，将同样的矢量在每个时间步骤输入，输出一个序列，图像生成标题：输入图片，得到和图片内容有关的标题；\n",
    "\n",
    "右下：编码器+解码器，编码器是sequence-to-vector，解码器是vector-to-sequence，翻译：先将一段话转化为矢量，再用解码器将矢量变为另一种语言。\n",
    "\n",
    "> 编码器+解码器由于sequence-to-sequence的原因：一句话的后面的词会影响到整句话的意思（比如日语总是在一句话的末尾表示否定），就像口译中有：交替传译vs同声传译（交替传译的效果由于同声传译，但是间隔长）\n",
    "\n",
    "## 训练RNN\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9dvlzy2j30ig0bjq53.jpg)\n",
    "\n",
    "## LSTM\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9fkkbdlj30ju0brq5r.jpg)\n",
    "\n",
    "## GRU\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9g378d2j30jv0d740e.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. NLP\n",
    "\n",
    "## 双向RNN\n",
    "\n",
    "一个RNN从左往右读，一个RNN从右往左读，在每个时间步骤，将输出叠加。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9iac0jkj30jr0c740s.jpg)\n",
    "\n",
    "\n",
    "```python\n",
    "keras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))\n",
    "```\n",
    "\n",
    "## 集束搜索\n",
    "\n",
    "\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g96szcs8oqj31o80u0tcq.jpg)\n",
    "\n",
    "\n",
    "\n",
    "Comment vas-tu?\n",
    "\n",
    "How are you?\n",
    "\n",
    "你好吗？\n",
    "\n",
    "\n",
    "\n",
    "## 注意力机制（2014）\n",
    "\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9j2wfnzj30l70d6dkm.jpg)\n",
    "\n",
    "\n",
    "除了将隐藏态传递给解码器，**也将编码器的所有输出传递给解码器**。\n",
    "\n",
    "对齐模型（或称为注意力层）产生权重*α*<sub>(*t*,*i*)</sub>\n",
    "\n",
    "注意力机制的目标是测量encoder的输出和decoder上一个隐藏态的相似度\n",
    "\n",
    "\n",
    "## 转换器（2017）\n",
    "\n",
    "Attention Is All You Need，没有使用RNN和CNN，只使用了注意力机制（加嵌入层、紧密层、归一化层。。。）\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9lcxup9j30ez0kmgpo.jpg)\n",
    "\n",
    "## 多头机制注意力\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/006y8mN6ly1g9a9mygy6jj30eb0iiq5v.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv_py3.7",
   "language": "python",
   "name": "venv_py3.7"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}