nn_language_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 运行环境"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-12-31T17:19:44+01:00\n",
      "\n",
      "CPython 3.6.2\n",
      "IPython 6.0.0\n",
      "\n",
      "compiler   : GCC 4.2.1 (Apple Inc. build 5666) (dot 3)\n",
      "system     : Darwin\n",
      "release    : 15.6.0\n",
      "machine    : x86_64\n",
      "processor  : i386\n",
      "CPU cores  : 4\n",
      "interpreter: 64bit\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import jieba\n",
    "import time\n",
    "from matplotlib import pyplot as plt\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.1.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建一个词的 Word Embedding\n",
    "在构建统计语言模型的时候，我们把所有看到的词都放入了 dict 里面，这时候影响不大，因为每个词占用的内存较小。\n",
    "\n",
    "在使用神经网络做自然语言处理的时候，我们一般都会对词表做一个截断操作，取最高频的 n 个（也有人按词频阈值做截断）。这样有两个好处：\n",
    "1. 减少模型的内存使用。\n",
    "2. 只出现过一两次的词，在整个优化过程中往往也很难学好。不如把这些词直接全看成未登录词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 决定了 embedding 的维度 （隐层节点数）\n",
    "word_embedding_dim = 128\n",
    "# 决定了词表数量, 预留一个未登录词\n",
    "vocab_size = 80000 + 1\n",
    "UNK_IDX = 0\n",
    "\n",
    "# 这里需要把 Word embedding 放到 Variable 里面。因为 Word embedding 是要随机初始化，跟着数据不断变化的。\n",
    "# 它相当于普通神经网络中的权重。\n",
    "\n",
    "# 在梯度下降时， tensorflow 的 Optimizer 会自动找到 Graph 中的 Variable，计算梯度并进行更新。\n",
    "word_embedding = tf.Variable(tf.random_uniform([vocab_size, word_embedding_dim]))\n",
    "\n",
    "# placeholder 和 variable 基本都可以当做 Tensor 来用\n",
    "# 注意这里的输入是 int32 类型，表示一个词 ID。这里我们需要对数据进行预处理，以把高频词映射到 [1, 80000] 之间，不在词表里面的词设置成 UNK, ID 为 0\n",
    "# 这里我们假设输入是两个词\n",
    "\n",
    "# 这里 Shape 的第一维我们指定为 None，是表示第一维可以根据数据进行变化，因此同样一个程序可以适应梯度下降时不同的 batch_size\n",
    "input_data = tf.placeholder(tf.int32, shape=[None, 2], name='input_data')\n",
    "\n",
    "input_embeds = tf.nn.embedding_lookup(word_embedding, input_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为 input_data 是一个二维矩阵，lookup 之后得到的其实是一个『三维的矩阵』\n",
    "\n",
    "怎么理解呢？如果我们一个样本有两个词，拿到的就是矩阵的两行，因此是一个矩阵\n",
    "\n",
    "但是我们同时有多个样本，因此需要对这个矩阵再『扩张』一个维度。而这个维度因为数据还未给出，大小是未知的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'embedding_lookup:0' shape=(?, 2, 128) dtype=float32>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_embeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 两个词的向量做相加"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reduce 开头的函数一般有一个 axis 参数，决定按行、按列或者按整个矩阵进行 reduce\n",
    "context_embeds = tf.reduce_sum(input_embeds, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Sum:0' shape=(?, 128) dtype=float32>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 注意观察 context_embds 的 shape\n",
    "# 因为 placeholder 第一位的维度是 None，这里 TF 没法确切知道第一维最后的 shape\n",
    "context_embeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相加的词向量再映射到 N 个词的概率分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 激活之前的输出\n",
    "raw_output = tf.layers.dense(context_embeds, vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = tf.nn.softmax(raw_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加 softmax 之后的输出\n",
    "final_output = tf.nn.softmax(raw_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1.30378403e-05   1.39868280e-05   1.24939406e-05   1.25732822e-05\n",
      "    1.20092827e-05   1.19113729e-05   1.33065150e-05   1.19917358e-05\n",
      "    1.17512154e-05   1.26534314e-05]]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    # 因为引入了 variable，所以需要进行初始化\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    # 输出的矩阵比较大，我们只看前 10 列\n",
    "    print(sess.run(final_output, feed_dict={input_data: np.asarray([[1, 2]])})[:, :10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 同第三讲示例代码中的类似 (sigmoid + cross entropy)，softmax 配合 cross entropy 的时候，在求导时两个连着看，也可以做分母的消除，因此在计算 cost 的时候我们要把 raw_output 喂给 tf 的这个损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 样本的 labels 也需要用 placeholder 放置\n",
    "labels = tf.placeholder(tf.int32, shape=[None], name='labels')\n",
    "\n",
    "# 因为我们每个样本的 label 只有一个，使用稠密的 softmax 算 cost 及求导太浪费了。这里使用 sparse 版本即可。\n",
    "# 如果你的 label 是完整的 N 个词上的概率分布，这时候可以使用 tf.nn.softmax_cross_entropy_with_logits\n",
    "cost = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=raw_output, labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0\n",
      "Cost: 9.917473\n",
      "Probability: 0.000012\n",
      "------\n",
      "Iteration 20\n",
      "Cost: 0.051485\n",
      "Probability: 0.000012\n",
      "------\n",
      "Iteration 40\n",
      "Cost: 0.020021\n",
      "Probability: 0.000012\n",
      "------\n",
      "Iteration 60\n",
      "Cost: 0.011911\n",
      "Probability: 0.000012\n",
      "------\n",
      "Iteration 80\n",
      "Cost: 0.009303\n",
      "Probability: 0.000012\n",
      "------\n"
     ]
    }
   ],
   "source": [
    "train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost)\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    dummy_feed_dict = {input_data: np.asarray([[1, 2]]),\n",
    "                       labels: np.asarray([3])}\n",
    "    for i in range(100):\n",
    "        sess.run(train_step, feed_dict=dummy_feed_dict)\n",
    "        if i % 20 == 0:\n",
    "            print(\"Iteration %d\" % i)\n",
    "            print(\"Cost: %f\" % cost.eval(feed_dict=dummy_feed_dict)[0])\n",
    "            # 查看输出中 ID == 3 的概率\n",
    "            print(\"Probability: %f\" % output.eval(session= sess,feed_dict=dummy_feed_dict)[0, 3])\n",
    "            print(\"------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练语料库，并完成 3 个名词各自最相近的 Top 10 个词的检索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_LM(filepath, ngram=1):\n",
    "    lm = defaultdict(Counter)\n",
    "    with open(filepath,'r') as f:\n",
    "        for line in f:\n",
    "            line = re.sub(r\"([%s]+)\" % u'。？！', r\"\\1\\n\", line.strip())\n",
    "            words = pseg.cut(line)\n",
    "            words = [i.word for i in words]\n",
    "            if len(words):\n",
    "                #为了统计句首的几个词，采用padding手段，在句首添加ngram个'<S>'\n",
    "                words = ['<S>'] * ngram + words\n",
    "                for i in range(ngram, len(words)):\n",
    "                    context = tuple(words[i-ngram : i])\n",
    "                    word = words[i]\n",
    "                    lm[context][word] += 1\n",
    "    return lm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('骑着马', 2), ('解开', 2), ('决心', 2), ('造物', 2), ('恶', 2)]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "read_size = 100000\n",
    "vocab_size = 2000 + 1  # 预留一个未登录词\n",
    "\n",
    "with open('一个分成两半的子爵.txt', encoding='utf-8') as f:\n",
    "    corpus = f.read(read_size) #建立语料库\n",
    "\n",
    "words = [word for word in jieba.cut(corpus) if word not in ' \\n']\n",
    "word_cnt = Counter(words)\n",
    "vocab = [i[0] for i in word_cnt.most_common(vocab_size - 1)]  # 词表(高频截断)\n",
    "\n",
    "# 将语料序列映射到 [0, vocab_size - 1] 内的整数, 未登录词为 0\n",
    "word_ids = [vocab.index(word) if (word in vocab) else 0 \n",
    "            for word in words]\n",
    "\n",
    "# 生成训练数据\n",
    "inputs_train = np.asarray([[word_ids[i-1], word_ids[i+1]] \n",
    "                           for i in range(1, len(word_ids) - 1)])\n",
    "labels_train = np.asarray(word_ids[1:-1])\n",
    "\n",
    "# 查看截断位置的词频\n",
    "word_cnt.most_common(vocab_size)[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46650, 30329, 2000, (30327, 2), (30327,))"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus), len(words), len(vocab), inputs_train.shape, labels_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "word_embedding_dim = 50\n",
    "word_embeddings = tf.Variable(tf.random_uniform([vocab_size, word_embedding_dim]))\n",
    "inputs = tf.placeholder(tf.int32, shape=[None, 2], name='inputs')\n",
    "labels = tf.placeholder(tf.int32, shape=[None], name='labels')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'embedding_lookup:0' shape=(?, 2, 50) dtype=float32>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_embeds = tf.nn.embedding_lookup(word_embeddings, inputs)\n",
    "input_embeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Sum:0' shape=(?, 50) dtype=float32>"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context_embeds = tf.reduce_sum(input_embeds, axis=1)\n",
    "context_embeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([Dimension(None), Dimension(2001)]), (30327,))"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_output = tf.layers.dense(context_embeds, vocab_size)\n",
    "output = tf.nn.softmax(raw_output)\n",
    "raw_output.shape, labels_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cost = tf.reduce_mean(\n",
    "    tf.nn.sparse_softmax_cross_entropy_with_logits(logits=raw_output, labels=labels))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([Dimension(None)]),\n",
       " (30327,),\n",
       " TensorShape([Dimension(None), Dimension(2)]),\n",
       " (30327, 2))"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.shape, labels_train.shape, inputs.shape, inputs_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 3\n",
    "epochs = 1000\n",
    "print_cost_every = 10\n",
    "batch_size = 100\n",
    "feed = {inputs: inputs_train, labels: labels_train}  # full-batch feed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000 - Cost: 7.61401\n",
      "Epoch 010 - Cost: 3.29275\n",
      "Epoch 020 - Cost: 2.90148\n",
      "Epoch 030 - Cost: 2.84078\n",
      "Epoch 040 - Cost: 2.84880\n",
      "Epoch 050 - Cost: 2.86327\n",
      "KeyboardInterrupt\n",
      "\n",
      "time: 173.71 s\n"
     ]
    }
   ],
   "source": [
    "#writer = tf.summary.FileWriter('/tensorflow/tf')\n",
    "cost_summary = tf.summary.scalar('cost_', cost)\n",
    "embedding_summary = tf.summary.histogram('embeddings_', word_embeddings)\n",
    "merged = tf.summary.merge_all()\n",
    "\n",
    "train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)\n",
    "\n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "costs = []\n",
    "\n",
    "# 训练集的随机序号\n",
    "num_inputs = len(labels_train)\n",
    "order = np.arange(num_inputs)\n",
    "np.random.shuffle(order)\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "try:\n",
    "    for i in range(epochs):\n",
    "        if i % print_cost_every == 0:\n",
    "            cost_value, merged_value = sess.run([cost, merged], feed_dict=feed)\n",
    "            print(\"Epoch {:03d} - Cost: {:.5f}\".format(i, cost_value))\n",
    "            costs.append(cost_value)\n",
    "          #  writer.add_summary(merged_value, i)\n",
    "        for j in range(0, num_inputs, batch_size):\n",
    "            batch_index = order[j: j + batch_size]\n",
    "            batch_inputs = inputs_train[batch_index]\n",
    "            batch_labels = labels_train[batch_index]\n",
    "            batch_feed = {inputs: batch_inputs, labels: batch_labels}\n",
    "            sess.run(train_step, feed_dict=batch_feed)\n",
    "except KeyboardInterrupt:\n",
    "    print('KeyboardInterrupt')\n",
    "finally:\n",
    "    end_time = time.time()\n",
    "    print('\\ntime: {:.2f} s'.format(end_time - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1550e56d8>]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plt.plot(costs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGC1JREFUeJzt3WtsW3ee3vHnR+pqWbZsSZbI+B479sQ2lWQ8SSZOsrvJ\nJHFierovZ4Et+qJA3mwXsyiK7e6bFl0UBYoWxfZFUSCY7aLFLHawnd0BajnjXDbZydi52kl8t2PZ\nsRNfZEm+25Isifz1BSlZVpyYsnl4ziG/H0AQRR5SD5H44dGPf55j7i4AQHwkwg4AAJgdihsAYobi\nBoCYobgBIGYobgCIGYobAGKG4gaAmKG4ASBmKG4AiJm6IB60o6PDly9fHsRDA0BV2rNnz5C7d5ay\nbSDFvXz5cu3evTuIhwaAqmRmp0rdllEJAMQMxQ0AMUNxA0DMUNwAEDMUNwDEDMUNADFDcQNAzESm\nuEfHc3rtvePa1TcUdhQAiLTIFHd9MqHX3vtSP/+w5DXoAFCTIlPcyYTplQ3deufIgK7fnAg7DgBE\nVmSKW5KymbRuTuT1j4fPhx0FACIrUsW9cdkCdc9r0ra958KOAgCRFaniTiRMWzIpvffFoK6MjIcd\nBwAiKVLFLUnZTEpjubzeOsS4BADuJHLF/ciSNi1e0Kxte8+GHQUAIilyxW1WGJfs6hvSpRtjYccB\ngMiJXHFL0tZMWhN5146D/WFHAYDIiWRxr0vP0/L2Oerdx7gEAGaKZHGbmbKZtD44fkGD126GHQcA\nIiWSxS1J2Z6U8i7tOMCabgCYLrLFvaarVasWzdW2fRQ3AEwX2eIujEtS+uTkRZ2/Ohp2HACIjMgW\nt1Q4dom7tJ29bgCYEuniXrVorr6XmsfqEgCYJtLFLRU+Av/pV5d15vJI2FEAIBIiX9xbM2lJ0nb2\nugFAUgyKe2n7HGUWz1cvc24AkBSD4pYK45J9p6/o5NCNsKMAQOhiUdxbJscl+9nrBoBYFPcDbc16\nbGkbh3oFAJVQ3Ga2xsw+n/Z11cz+pBLhpstm0jrSf019A9cr/asBIFLuWtzuftTdH3H3RyR9X9Kw\npF8FnmyGLZmUzMSabgA1b7ajkuclHXf3U0GE+S5d85r0g+UL1bvvnNy90r8eACJjtsX9E0l/G0SQ\nUmzNpNQ3cF1Hz18LKwIAhK7k4jazBkk/lvR/v+X2V81st5ntHhwcLFe+22xen1LCpN69rC4BULtm\ns8f9sqRP3f2Op19399fcfaO7b+zs7CxPuhk6Wxv11IMd6t13lnEJgJo1m+L+A4U4JpmUzaR08sKw\nDp69GnYUAAhFScVtZi2SXpD0D8HGubvN67tVlzBtY3UJgBpVUnG7+w13b3f3K0EHupu2OQ16enWH\neveyugRAbYrFJydnymbSOnN5RJ99fTnsKABQcbEs7hfXdakhmWB1CYCaFMvintdUr2cf6tTr+88p\nn2dcAqC2xLK4JWlrT0r9V0e1+9SlsKMAQEXFtrif/16XGusSHLsEQM2JbXHPbazTc2sX6fX9/cox\nLgFQQ2Jb3FJhdcnQ9Zv66MSFsKMAQMXEurifW7tIcxqS2sb5KAHUkFgXd3NDUj/6Xpd2HDin8Vw+\n7DgAUBGxLm6pcOySS8Pjev844xIAtSH2xf07azrV2linXs5HCaBGxL64G+uSemFdl9442K+xCcYl\nAKpf7ItbkrZm0ro6OqHfHgvmBA4AECVVUdybVnVofnO9tjEuAVADqqK4G+oS2ryuW28dOq/R8VzY\ncQAgUFVR3JKU7UnpxlhO/3R0IOwoABCoqinuH65sV3tLAx/GAVD1qqa465IJbV7frXcOD2h4bCLs\nOAAQmKopbqlw7JKR8Zz+8TDjEgDVq6qK+/EVC9XZ2sihXgFUtaoq7mTCtGVDSu8eHdS10fGw4wBA\nIKqquKXCmXHGJvJ6+/D5sKMAQCCqrrgfXbJA6flNnEgYQNWquuJOJExbMim9d2xQV4YZlwCoPlVX\n3FJhdcl4zvXGof6wowBA2VVlcWcWz9fShXM4dgmAqlSVxW1WGJe8f/yCLly/GXYcACirqixuqXBm\nnFzeteMg4xIA1aVqi/vh1Dyt7GhhdQmAqlO1xW1mymZS+ujLCxq4Nhp2HAAom6otbknK9qSVd+nX\n+xmXAKgeVV3cD3W16qGuuRy7BEBVKam4zazNzH5pZkfM7LCZ/TDoYOWyNZPWJycv6dyVkbCjAEBZ\nlLrH/d8l7XD3tZJ6JB0OLlJ5ZXvSkqTtnGABQJW4a3Gb2XxJz0r6K0ly9zF3vxx0sHJZ0dGidel5\n6qW4AVSJUva4V0galPTXZvaZmf3MzFoCzlVW2Uxan399WV9fHA47CgDct1KKu07SY5L+p7s/KumG\npD+buZGZvWpmu81s9+DgYJlj3p9sJiVJ2r6fvW4A8VdKcZ+WdNrdPyr+/EsVivw27v6au290942d\nnZ3lzHjfliyco54lbawuAVAV7lrc7t4v6WszW1O86nlJhwJNFYCtmZQOnLmqL4duhB0FAO5LqatK\n/ljS35jZPkmPSPpPwUUKxisbCuOSXo4YCCDmSipud/+8OAbJuPvvu/uloIOVW7qtWRuXLWB1CYDY\nq+pPTs6UzaR09Pw1HTt/LewoAHDPaqq4X9mQkpm0jb1uADFWU8W9aF6TnlixUL37zsrdw44DAPek\npopbkrb2pHVi8IYOn2NcAiCeaq64X16fUjJhrOkGEFs1V9wLWxr01IPt6t13jnEJgFiqueKWCod6\n/erisPafuRJ2FACYtZos7pfWdas+aazpBhBLNVnc8+fU65nVndrOuARADNVkcUuFD+OcuTyiT7+K\nzaHFAUBSDRf3Cw93qaEuoW0cuwRAzNRscbc21et3H+rU6/vPKZdnXAIgPmq2uKXC+SgHrt3UJycv\nhh0FAEpW08X9/NpFaqpP8GEcALFS08Xd0lin59d26df7+zWRy4cdBwBKUtPFLUlbe1K6cGNMH55g\nXAIgHmq+uH93zSK1NCQZlwCIjZov7qb6pF54uEs7DvZrnHEJgBio+eKWpGwmrcvD49rZNxR2FAC4\nK4pb0jMPdai1qU69ezl2CYDoo7glNdYl9dK6br15qF83J3JhxwGA70RxF2UzKV0bndB7XzAuARBt\nFHfRplUdWjCnnmOXAIg8iruoPpnQ5vXdevvweY2MMS4BEF0U9zTZTFrDYzm9e3Qg7CgA8K0o7mme\nWLFQHXMb+DAOgEijuKepSyb08vqU3jkyoBs3J8KOAwB3RHHPsLUnrdHxvN4+fD7sKABwRxT3DBuX\nLVD3vCZOJAwgsijuGRIJ0ysbUvrN0UFdHR0POw4AfAPFfQfZnpTGcnm9dZBxCYDoobjv4NElbXqg\nrZnVJQAiieK+AzNTNpPSb48N6fLwWNhxAOA2JRW3mZ00s/1m9rmZ7Q46VBRkM2lN5F1vHOwPOwoA\n3GY2e9y/5+6PuPvGwNJEyPoH5mlZ+xxWlwCIHEYl32JyXLKrb0hD12+GHQcAppRa3C7pTTPbY2av\n3mkDM3vVzHab2e7BwcHyJQxRNpNW3qVfH2BcAiA6Si3up939MUkvS/ojM3t25gbu/pq7b3T3jZ2d\nnWUNGZa13a16sLNFvRzqFUCElFTc7n6m+H1A0q8kPR5kqKgojEvS+vjkRZ2/Ohp2HACQVEJxm1mL\nmbVOXpb0oqQDQQeLiq09KblLr+/nTUoA0VDKHneXpJ1mtlfSx5K2u/uOYGNFx6pFrVrb3crqEgCR\nUXe3Ddz9hKSeCmSJrK09af2XN47q7OURpduaw44DoMaxHLAE2UxKkrSdvW4AEUBxl2BZe4s2PDCf\nY5cAiASKu0TZTEp7T1/RVxeGw44CoMZR3CXaUhyX9O5nrxtAuCjuEi1eMEePLm1T717m3ADCRXHP\nQjaT1qFzV3V88HrYUQDUMIp7FrZsSMlM7HUDCBXFPQvd85v0g2ULWV0CIFQU9yxle1I6NnBdR/uv\nhR0FQI2iuGfp5fUpJUzsdQMIDcU9S52tjfrhg+3q3XdO7h52HAA1iOK+B9lMWl8O3dDBs1fDjgKg\nBlHc92Dzum7VJYwjBgIIBcV9Dxa0NGjTqg717jvLuARAxVHc9yibSen0pRHtPX0l7CgAagzFfY9e\nXNethmSC81ECqDiK+x7Nb67Xsw91aPv+c8rnGZcAqByK+z5kM2mduzKqT7+6FHYUADWE4r4PP3q4\nS411CW1jXAKggiju+zC3sU6/t2aRXj/QrxzjEgAVQnHfp2xPSoPXbuqjLy+EHQVAjaC479Nzaxep\nuT7Jh3EAVAzFfZ/mNNTpRw93aceBfk3k8mHHAVADKO4yyGZSunhjTO8fZ1wCIHgUdxn8zkOdam2s\n41CvACqC4i6DpvqkXiiOS8YmGJcACBbFXSbZnpSujk5oZ99g2FEAVDmKu0yeXtWp+c31nEgYQOAo\n7jJpqEvopXVdevPQeY2O58KOA6CKUdxllM2kdf3mhH7zBeMSAMGhuMvoqQfbtbClgQ/jAAgUxV1G\ndcmENq/v1tuHzmt4bCLsOACqVMnFbWZJM/vMzHqDDBR32UxKI+M5vXNkIOwoAKrUbPa4fyrpcFBB\nqsUTK9rV2drI6hIAgSmpuM1ssaQtkn4WbJz4SyZMWzak9O7RAV2/ybgEQPmVusf9l5L+VBIfCyxB\nNpPSzYm83j50PuwoAKrQXYvbzLKSBtx9z122e9XMdpvZ7sHB2l4O99jSBUrNb+LYJQACUcoe9yZJ\nPzazk5J+Iek5M/v5zI3c/TV33+juGzs7O8scM14SxXHJb74Y1JWR8bDjAKgydy1ud/9zd1/s7ssl\n/UTSO+7+h4Eni7lsT1rjOdebB/vDjgKgyrCOOyA9i+drycJmPowDoOxmVdzu/k/ung0qTDUxM23Z\nkNauviFdujEWdhwAVYQ97gBlMylN5F07GJcAKCOKO0Dr0vO0oqOF1SUAyoriDpCZKZtJ6YPjFzR4\n7WbYcQBUCYo7YNlMWnmXfn2ANykBlAfFHbA13a1avWguxy4BUDYUdwVs7Unrk1MX1X9lNOwoAKoA\nxV0B2UxK7tL2/ex1A7h/FHcFrOycq4dT81hdAqAsKO4Kyfak9NlXl3X60nDYUQDEHMVdIdkNaUnS\ndj4CD+A+UdwVsrR9jnoWz+fYJQDuG8VdQdlMWvvPXNHJoRthRwEQYxR3BW3JpCSxugTA/aG4Kyjd\n1qzvL1ugbXtZXQLg3lHcFZbNpHSk/5r6Bq6FHQVATFHcFfbKhpTMpG18BB7APaK4K6xrXpMeX75Q\nvfvOyt3DjgMghijuEGztSev44A0d6WdcAmD2KO4QvLy+W8mE8RF4APeE4g5B+9xGPfVgu3r3nWNc\nAmDWKO6QZDMpnbowrANnroYdBUDMUNwheWldt+oYlwC4BxR3SNrmNOiZ1R2MSwDMGsUdomwmrTOX\nR/TXu05q4BpnxwFQmrqwA9SyF9d1acW7LfqL3kP6i95DWtPVqk2rOvT06nY9saJdLY385wHwTRbE\nn+kbN2703bt3l/1xq1Eu7zp09qp29g1pV9+QPj55UWMTedUlTI8tXTBV5JnFbapP8gcSUK3MbI+7\nbyxpW4o7WkbHc9pz6pJ29g1p57EhHTh7Re7S3MY6PblyYaHIV3Vo1aK5MrOw4wIoE4q7ily6MaYP\nTlyY2iM/daFw6rOueY1TJb5pVYe65jWFnBTA/aC4q9jXF4e1q29Iv+0b0vt9Q7o0PC5JWr1orjat\n6tAzqzv0xMp2zWU+DsQKxV0j8nnXoXNXtatvSDv7hvTxlxd1szgff2RJW3E+3qFHljAfB6KO4q5R\no+M5ffrVpUKRHxvSvjOF+XhLQ1JPrGyf2iNfzXwciByKG5KkK8Pj+uDEUHE+fkFfFs912dnaODUb\n37SqXan5zSEnBUBx445OXxoujlUu6P2+IV24MSZJerCzRU+v6tDTqzv1xMqFmtdUH3JSoPaUtbjN\nrEnSe5IaVfjAzi/d/d9/130o7ujL511H+q9Nzcc/+vKCRsfzSiZMPYvnT+2RP7p0gRrqmI8DQSt3\ncZukFne/bmb1knZK+qm7f/ht96G44+fmRE6ffXVZO48Vinzf6cvKuzSnIanHVyws7pF3aE1XK/Nx\nIACzKe67rhnzQrNfL/5YX/ziqEhVprEuqSdXtuvJle36Ny+t0ZWRcX144sLUHvl/3H5YktQxt1Gb\nVrVPrSFPtzEfByqtpBm3mSUl7ZG0StL/cPd/e4dtXpX0qiQtXbr0+6dOnSpzVITp7OWRqQ8B7eob\n0tD1wnx8ZUfL1LLDJ1e2a34z83HgXgT25qSZtUn6laQ/dvcD37Ydo5Lq5u46ev6adh4rlPhHX17U\n8FhOCZMyi9um5uOPLWtTY10y7LhALAS6qsTM/p2kYXf/r9+2DcVdW8Ym8vpscv1435D2nr6iXN7V\nVJ/Q4yva9UyxyNd2tyqRYD4O3Em535zslDTu7pfNrFnSm5L+s7v3ftt9KO7adnV0XB+duDhV5H0D\nhbdI2lsa9NSqDj25cqE65jZqTkNSzfVJNTckNaehbupyc32SlSyoOWV9c1JSStL/Ls65E5L+7rtK\nG5jXVK8XHu7SCw93SZL6r4xOzcd39g1p2967n66tLmHTSj2ppvrC90Kx1xWunyz6mZenXhAKLwbT\n7z95ubEuweoYxBYfwEFFubu+vjiiq6PjGhnPaWQsp+GxnEbGJzQyltfw2IRGxyevm3574fLI5G1j\nE1OXR8dzGs/N7v/jhGmq3Kfv+U8VffHF4NblOjU3JG57MZi+/cy/HJrqo/PC4O7Ke+G7S8q7y12F\nLxUu54u3ef6b1+ULG95238nvmnmdio9bvJzLu/LuyucLt+fclc8X8uTyLp+8zlW83ov3KWw/+bP7\ntMdyVy5/6/Z83pUr/s7b7pu/w2OX9Pv1Hblu3ffW75dyXnis+c31+tm/+ME9/Xcq9x43UDZmpqXt\nc8r+uOO5/O1FP+PF4LteBAovFBNTl/uvznxRyWlsIj/rTNP39pvqC6OfYgfeKlMV/uFr5nXFYpVu\nle708nW/ffvpRTxzOxReqJMJk5kpaaaESYmEKZkwJWzyS7d+TkiJ4rY2/fribUkzJYrXTW5Tl0xU\nbMRHcaMq1CcTqk8mAvu4/kQur9GJ4l8EY3kNjxeLvoS/CEbGc7o5npdMMhUKwaZdlhWvk2R263ap\nUCZ22+2T97112x2vK15WsZBMVnyc27efuq542Yq/Z3K7yceYnnty++m5E4lvPobZjIJLaFpxfrMA\nbyvOabebTRasitcXH2Naud5Wtgm79buL21cbihsoQV0yobnJBMc5RyTw1j0AxAzFDQAxQ3EDQMxQ\n3AAQMxQ3AMQMxQ0AMUNxA0DMUNwAEDOBHKvEzAYl3euZFDokDZUxThzwnKtfrT1fiec8W8vcvbOU\nDQMp7vthZrtLPdBKteA5V79ae74SzzlIjEoAIGYobgCImSgW92thBwgBz7n61drzlXjOgYncjBsA\n8N2iuMcNAPgOkSluM9tsZkfNrM/M/izsPJVgZv/LzAbM7EDYWSrBzJaY2btmdsjMDprZT8POFDQz\nazKzj81sb/E5/4ewM1WKmSXN7DMzq4lz1JrZSTPbb2afm1mg526MxKikeCLiLyS9IOm0pE8k/YG7\nHwo1WMDM7FlJ1yX9H3dfH3aeoJlZSlLK3T81s1ZJeyT9fjX/d7bCiSdb3P26mdVL2inpp+7+YcjR\nAmdm/1rSRknz3D0bdp6gmdlJSRvdPfC161HZ435cUp+7n3D3MUm/kPTPQs4UOHd/T9LFsHNUiruf\nc/dPi5evSTos6YFwUwXLC64Xf6wvfoW/txQwM1ssaYukn4WdpRpFpbgfkPT1tJ9Pq8r/Qdc6M1su\n6VFJH4WbJHjFkcHnkgYkveXuVf+cJf2lpD+VNPuzLMeXS3rTzPaY2atB/qKoFDdqiJnNlfT3kv7E\n3a+GnSdo7p5z90ckLZb0uJlV9VjMzLKSBtx9T9hZKuxpd39M0suS/qg4Cg1EVIr7jKQl035eXLwO\nVaY45/17SX/j7v8Qdp5KcvfLkt6VtDnsLAHbJOnHxZnvLyQ9Z2Y/DzdS8Nz9TPH7gKRfqTACDkRU\nivsTSavNbIWZNUj6iaT/F3ImlFnxjbq/knTY3f9b2Hkqwcw6zayteLlZhTfgj4SbKlju/ufuvtjd\nl6vwb/kdd//DkGMFysxaim+4y8xaJL0oKbDVYpEobnefkPSvJL2hwhtWf+fuB8NNFTwz+1tJH0ha\nY2anzexfhp0pYJsk/XMV9sA+L369EnaogKUkvWtm+1TYQXnL3WtieVyN6ZK008z2SvpY0nZ33xHU\nL4vEckAAQOkisccNACgdxQ0AMUNxA0DMUNwAEDMUNwDEDMUNADFDcQNAzFDcABAz/x9uvCR7/DGI\n7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b000eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "validate_words = ['解开', '决心', '造物']\n",
    "validate_ids = [vocab.index(word) for word in validate_words]\n",
    "validate_inputs = tf.constant(validate_ids, dtype=tf.int32)\n",
    "\n",
    "norm = tf.sqrt(tf.reduce_sum(tf.square(word_embeddings), 1, keep_dims=True))\n",
    "normalized_embeddings = word_embeddings / norm\n",
    "validate_embeddings = tf.nn.embedding_lookup(normalized_embeddings, validate_inputs)\n",
    "similarity = tf.matmul(validate_embeddings, normalized_embeddings, transpose_b=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "similar to 解开:\n",
      "['不行', '再次', '茅草', '前去', '莱说', '想起', '黑衣服', '肩', '给予', '出去']\n",
      "similar to 决心:\n",
      "['前去', '不行', '颜色', '蹦跳', '喜鹊', '酒', '菲奥', '文学', '为的是', '体质']\n",
      "similar to 造物:\n",
      "['全身', '敏锐', '盛大', '本性', '年岁', '吩咐', '尸体', '气', '残缺不全', '居民']\n"
     ]
    }
   ],
   "source": [
    "sim_values = sess.run(similarity, feed_dict=feed)\n",
    "for i in range(len(validate_words)):\n",
    "    word = validate_words[i]\n",
    "    similar_ids = (-sim_values[i, :]).argsort()[1: 11]\n",
    "    similar_words = [vocab[j] for j in similar_ids]\n",
    "    print('similar to {}:'.format(word))\n",
    "    print(similar_words)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}