注意力机制.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 注意力机制\n",
    "***\n",
    "***\n",
    "Time: 2020-09-16<br>\n",
    "Author: dsy<br>\n",
    "Notes: 《神经网络与深度学习》\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自注意力"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设输入序列为$X=[x_1,\\cdots,x_N] \\in \\mathbb{R}^{d_1 x N}$，输出序列为$H=[h_1,\\cdots,h_N] \\in \\mathbb{R}^{d_2 x N}$,首先我们可以通过线性变换得到三组向量序列：\n",
    "$$\n",
    "\\begin{aligned}\n",
    "Q & = W_Q X \\in \\mathbb{R}^{d_3 x N},\\\\\n",
    "K & = W_K X \\in \\mathbb{R}^{d_3 x N},\\\\\n",
    "V & = W_V X \\in \\mathbb{R}^{d_2 x N},\n",
    "\\end{aligned}\n",
    "$$\n",
    "其中$Q,K,V$分别为查询向量序列，键向量序列和值向量序列，$W_Q \\in \\mathbb{R}^{d_3 x d_1},W_K \\in \\mathbb{R}^{d_3 x d_1},W_V \\in \\mathbb{R}^{d_2 x d_1}$分别为可学习的参数矩阵。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用公式可以得出输出向量$h_i$:\n",
    "$$\n",
    "\\begin{aligned}\n",
    "h_i & = att \\Big( (K,V),q_i\\Big) \\\\\n",
    "    & = \\sum_{j=1}^{N} \\alpha_{ij}v_j\\\\\n",
    "    & = \\sum_{j=1}^{N} softmax \\Big(s(k_j,q_i) \\Big)v_j\n",
    "\\end{aligned}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "常用的注意力打分函数：\n",
    "$$\\begin{array}{cll}\n",
    "\\text{加性模型} & s(x_i,q) & = v^T \\tanh(W x_i + U q) \\\\\n",
    "\\text{点积模型} & s(x_i,q) & = x_i^Tq \\\\\n",
    "\\text{缩放点积模型} & s(x_i,q) & = \\frac{x_i^T q}{\\sqrt{d}} \\\\\n",
    "\\text{双线性模型} & s(x_i,q) & = x_i^T W q \n",
    "\\end{array}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1dad61b22d0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "seed = 0\n",
    "torch.manual_seed(seed)\n",
    "torch.random.manual_seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionFromDsy(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(AttentionFromDsy,self).__init__()\n",
    "        \n",
    "    def s(self,k,q,d):\n",
    "        return (k.T.reshape((1,-1))).mm(q.reshape((-1,1))) / torch.sqrt(torch.Tensor([d]))\n",
    "        \n",
    "    def forward(self,X):\n",
    "        d1 ,N = X.shape\n",
    "        d2,d3 = 10,10\n",
    "        \n",
    "        h = torch.zeros((d2,N))\n",
    "        \n",
    "        WQ = torch.randn((d3,d1),requires_grad=True) # d3 ,d1\n",
    "        WK = torch.randn((d3,d1),requires_grad=True) # d3,d1\n",
    "        WV = torch.randn((d2,d1),requires_grad=True) # d2 ,d1\n",
    "        \n",
    "        Q = WQ.mm(X) # d3,N\n",
    "        K = WK.mm(X) # d3,N\n",
    "        V = WV.mm(X) # d2,N\n",
    "        \n",
    "        for i in range(N):\n",
    "            sha = [self.s(K[:,j],Q[:,i],d3) for j in range(N)]\n",
    "            sha_softmax =  torch.nn.functional.softmax(torch.Tensor(sha))\n",
    "            shaaa = torch.zeros((d2,))\n",
    "            for j in range(N):\n",
    "                sha_softmax_j = sha_softmax[j]\n",
    "                shaaa += sha_softmax_j * V[:,j]\n",
    "            h[:,i] = shaaa\n",
    "            \n",
    "        return h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.1258, -1.1524, -0.2506, -0.4339],\n",
       "        [ 0.8487,  0.6920, -0.3160, -2.1152],\n",
       "        [ 0.3223, -1.2633,  0.3500,  0.3081],\n",
       "        [ 0.1198,  1.2377,  1.1168, -0.2473]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = torch.randn((4,4))\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\dsy\\desktop\\deep_learning_from_dsy\\venv\\lib\\site-packages\\ipykernel_launcher.py:24: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.5689,  0.6598,  1.5184,  0.9254],\n",
       "        [-0.2293, -0.4523, -2.2452, -1.1773],\n",
       "        [-0.7210, -0.8249, -0.7496, -1.3198],\n",
       "        [-0.2952, -0.1535, -0.0277,  0.3204],\n",
       "        [ 0.9266,  0.9983,  1.2646,  1.3426],\n",
       "        [-0.2005, -0.2239, -0.0952, -0.3015],\n",
       "        [ 1.1224,  1.0853, -0.0742,  1.0859],\n",
       "        [ 0.8179,  0.6819,  0.3794, -0.0328],\n",
       "        [ 0.1456,  0.1609,  1.5256,  0.2119],\n",
       "        [ 0.1954, -0.0705, -0.0035, -1.1535]], grad_fn=<CopySlices>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "afd = AttentionFromDsy()\n",
    "afd(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "L,N,E,S = 3,4,10,6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "multiheadAttention = torch.nn.MultiheadAttention(\n",
    "    embed_dim = E\n",
    "    , num_heads=10\n",
    "    , dropout=0.0\n",
    "    , bias=True\n",
    "    , add_bias_kv=False\n",
    "    , add_zero_attn=False\n",
    "    , kdim=None\n",
    "    , vdim=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 4, 10])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = torch.rand((L,N,E))\n",
    "query.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 4, 10])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = torch.rand((S,N,E))\n",
    "key.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 4, 10])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "value = torch.rand((S,N,E))\n",
    "value.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "attn_output,attn_output_weights = multiheadAttention(\n",
    "    query\n",
    "    , key\n",
    "    , value\n",
    "    , key_padding_mask=None\n",
    "    , need_weights=True\n",
    "    , attn_mask=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 4, 10])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attn_output.shape # L,N,E"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 3, 6])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attn_output_weights.shape # N,L,S"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}