Extra - Document-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document embedding (doc2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general idea here is to train a neural network model to create text embeddings.\n",
    "Text embedding vectors can then be used to obtrain similarity metrics on neighbouring texts.\n",
    "\n",
    "In this example we use the `doc2vec` model \n",
    "([Le & Mikolov, ICML 2014](https://cs.stanford.edu/~quocle/paragraph_vector.pdf))\n",
    "to create text embeddings using pubmed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import re\n",
    "import os\n",
    "import collections\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import gensim\n",
    "from gensim.models import Doc2Vec\n",
    "from gensim.models.doc2vec import TaggedDocument\n",
    "\n",
    "import config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- word2vec -> doc2vec\n",
    "  - skip-gram -> distributed memory\n",
    "  - continuous-bag-of-words -> distributed bag-of-words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Distributed Memory model (PV-DM)](https://adriancolyer.files.wordpress.com/2016/05/paragraph-vectors-fig-2.png?w=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Distributed Bag of Words model (PV-DBW)](https://adriancolyer.files.wordpress.com/2016/05/paragraph-vectors-fig-3.png?w=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-processing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the pubmed data from earlier sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pmid</th>\n",
       "      <th>year</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25475436</td>\n",
       "      <td>2015</td>\n",
       "      <td>Sixty-five common genetic variants and predict...</td>\n",
       "      <td>We developed a 65 type 2 diabetes (T2D) varian...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25011450</td>\n",
       "      <td>2014</td>\n",
       "      <td>Association between alcohol and cardiovascular...</td>\n",
       "      <td>To use the rs1229984 variant in the alcohol de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28968714</td>\n",
       "      <td>2018</td>\n",
       "      <td>FATHMM-XF: accurate prediction of pathogenic p...</td>\n",
       "      <td>We present FATHMM-XF, a method for predicting ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21965548</td>\n",
       "      <td>2012</td>\n",
       "      <td>Four genetic loci influencing electrocardiogra...</td>\n",
       "      <td>Presence of left ventricular hypertrophy on an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>26930047</td>\n",
       "      <td>2016</td>\n",
       "      <td>Diagnosis of Coronary Heart Diseases Using Gen...</td>\n",
       "      <td>Cardiovascular disease (including coronary art...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       pmid  year                                              title  \\\n",
       "0  25475436  2015  Sixty-five common genetic variants and predict...   \n",
       "1  25011450  2014  Association between alcohol and cardiovascular...   \n",
       "2  28968714  2018  FATHMM-XF: accurate prediction of pathogenic p...   \n",
       "3  21965548  2012  Four genetic loci influencing electrocardiogra...   \n",
       "4  26930047  2016  Diagnosis of Coronary Heart Diseases Using Gen...   \n",
       "\n",
       "                                            abstract  \n",
       "0  We developed a 65 type 2 diabetes (T2D) varian...  \n",
       "1  To use the rs1229984 variant in the alcohol de...  \n",
       "2  We present FATHMM-XF, a method for predicting ...  \n",
       "3  Presence of left ventricular hypertrophy on an...  \n",
       "4  Cardiovascular disease (including coronary art...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pubmed_data = pd.read_csv(config.demoPubmedFile, sep=\"\\t\")\n",
    "\n",
    "pubmed_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some of the pre-processing steps to be done (subjective):\n",
    "\n",
    "- Make sure `abstract`s are strings.\n",
    "- Keep only abstracts that are sufficiently long enough, and not just fragments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def long_enough(text, word_length=40, num_sentences=2):\n",
    "    return (len(text.split(\" \")) >= word_length \n",
    "            and text.count(\".\") >= num_sentences)\n",
    "\n",
    "pubmed_data = pubmed_data \\\n",
    "    .assign(abstract=lambda df: df.abstract.astype(str)) \\\n",
    "    .assign(keep=lambda df: df.abstract.apply(long_enough))\n",
    "\n",
    "pubmed_data_keep = pubmed_data.query(\"keep\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use the rs1229984 variant in the alcohol dehydrogenase 1B gene (ADH1B) as an instrument to investigate the causal role of alcohol in cardiovascular disease. \n",
      "\n",
      "Conclusions. A candidate functional variant, rs28451064, was identified. Future work should focus on identifying the pathway(s) involved. \n",
      "\n",
      "Haptoglobin acts as an antioxidant by limiting peroxidative tissue damage by free hemoglobin. The haptoglobin gene allele Hp2 comprises a 1.7 kb partial duplication. Relative to allele Hp1, Hp2 carriers form protein multimers, suboptimal for hemoglobin scavenging. \n",
      "\n",
      "To establish whether the association between milk intake and prostate cancer operates via the insulin-like growth factor (IGF) pathway (including IGF-I, IGF-II, IGFBP-1, IGFBP-2, and IGFBP-3). \n",
      "\n",
      "Prenatal exposure to maternal cigarette smoking (prenatal smoke exposure) had been associated with altered DNA methylation (DNAm) at birth. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Abstracts to be discarded from the corpus\n",
    "for text in pubmed_data.query(\"not keep\").abstract[:5]:\n",
    "    print(text, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create two sets of text corpus, in which a text element consists of both the `title` and the `abstract`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of texts in training set: 8379\n",
      "Number of texts in test set: 441\n"
     ]
    }
   ],
   "source": [
    "split = 0.95\n",
    "split_idx = int(np.floor(pubmed_data_keep.shape[0] * split))\n",
    "\n",
    "pubmed_train = pubmed_data_keep[:split_idx]\n",
    "pubmed_test = pubmed_data_keep[split_idx:]\n",
    "train_corpus = []\n",
    "test_corpus = []\n",
    "\n",
    "for i, (title, abstract) in enumerate(zip(pubmed_train.title, \n",
    "                                          pubmed_train.abstract)):\n",
    "    train_corpus.append(TaggedDocument(title + \" \" + abstract, [i]))\n",
    "    \n",
    "for i, (title, abstract) in enumerate(zip(pubmed_test.title,\n",
    "                                          pubmed_test.abstract)):\n",
    "    test_corpus.append(TaggedDocument(title + \" \" + abstract, [i]))\n",
    "    \n",
    "print(f\"Number of texts in training set: {len(train_corpus)}\")\n",
    "print(f\"Number of texts in test set: {len(test_corpus)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TaggedDocument(Sixty-five common genetic variants and prediction of type 2 diabetes. We developed a 65 type 2 diabetes (T2D) variant-weighted gene score to examine the impact on T2D risk assessment in a U.K.-based consortium of prospective studies, with subjects initially free from T2D (N = 13,294; 37.3% women; mean age 58.5 [38-99] years). We compared the performance of the gene score with the phenotypically derived Framingham Offspring Study T2D risk model and then the two in combination. Over the median 10 years of follow-up, 804 participants developed T2D. The odds ratio for T2D (top vs. bottom quintiles of gene score) was 2.70 (95% CI 2.12-3.43). With a 10% false-positive rate, the genetic score alone detected 19.9% incident cases, the Framingham risk model 30.7%, and together 37.3%. The respective area under the receiver operator characteristic curves were 0.60 (95% CI 0.58-0.62), 0.75 (95% CI 0.73 to 0.77), and 0.76 (95% CI 0.75 to 0.78). The combined risk score net reclassification improvement (NRI) was 8.1% (5.0 to 11.2; P = 3.31 × 10(-7)). While BMI stratification into tertiles influenced the NRI (BMI ≤24.5 kg/m(2), 27.6% [95% CI 17.7-37.5], P = 4.82 × 10(-8); 24.5-27.5 kg/m(2), 11.6% [95% CI 5.8-17.4], P = 9.88 × 10(-5); >27.5 kg/m(2), 2.6% [95% CI -1.4 to 6.6], P = 0.20), age categories did not. The addition of the gene score to a phenotypic risk model leads to a potentially clinically important improvement in discrimination of incident T2D. , [0]) \n",
      "\n",
      "TaggedDocument(FATHMM-XF: accurate prediction of pathogenic point mutations via extended features. We present FATHMM-XF, a method for predicting pathogenic point mutations in the human genome. Drawing on an extensive feature set, FATHMM-XF outperforms competitors on benchmark tests, particularly in non-coding regions where the majority of pathogenic mutations are likely to be found., [1]) \n",
      "\n",
      "TaggedDocument(Four genetic loci influencing electrocardiographic indices of left ventricular hypertrophy. Presence of left ventricular hypertrophy on an ECG (ECG-LVH) is widely assessed clinically and provides prognostic information in some settings. There is evidence for significant heritability of ECG-LVH. We conducted a large-scale gene-centric association analysis of 4 commonly measured indices of ECG-LVH., [2]) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(3):\n",
    "    print(train_corpus[i], \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a Doc2Vec model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we demonstrate a simple usage of paragraph / sentence embedding using a Doc2Vec model.\n",
    "\n",
    "Refer to [gensim's documentation](https://radimrehurek.com/gensim/models/doc2vec.html) \n",
    "on the specific usage of Doc2Vec model and its APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 28s, sys: 8.01 s, total: 3min 36s\n",
      "Wall time: 1min 33s\n"
     ]
    }
   ],
   "source": [
    "d2v_model = Doc2Vec(train_corpus)\n",
    "\n",
    "%time d2v_model.train(train_corpus, total_examples=d2v_model.corpus_count, epochs=40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m\n",
       "\u001b[0mDoc2Vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdocuments\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdm_mean\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdm\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdbow_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdm_concat\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdm_tag_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdocvecs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdocvecs_mapfile\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mcomment\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtrim_rule\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m      Class for training, using and evaluating neural networks described in http://arxiv.org/pdf/1405.4053v2.pdf\n",
       "\u001b[0;31mInit docstring:\u001b[0m\n",
       "Initialize the model from an iterable of `documents`. Each document is a\n",
       "TaggedDocument object that will be used for training.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "documents : iterable of iterables\n",
       "    The `documents` iterable can be simply a list of TaggedDocument elements, but for larger corpora,\n",
       "    consider an iterable that streams the documents directly from disk/network.\n",
       "    If you don't supply `documents`, the model is left uninitialized -- use if\n",
       "    you plan to initialize it in some other way.\n",
       "\n",
       "dm : int {1,0}\n",
       "    Defines the training algorithm. If `dm=1`, 'distributed memory' (PV-DM) is used.\n",
       "    Otherwise, `distributed bag of words` (PV-DBOW) is employed.\n",
       "\n",
       "size : int\n",
       "    Dimensionality of the feature vectors.\n",
       "window : int\n",
       "    The maximum distance between the current and predicted word within a sentence.\n",
       "alpha : float\n",
       "    The initial learning rate.\n",
       "min_alpha : float\n",
       "    Learning rate will linearly drop to `min_alpha` as training progresses.\n",
       "seed : int\n",
       "    Seed for the random number generator. Initial vectors for each word are seeded with a hash of\n",
       "    the concatenation of word + `str(seed)`. Note that for a fully deterministically-reproducible run,\n",
       "    you must also limit the model to a single worker thread (`workers=1`), to eliminate ordering jitter\n",
       "    from OS thread scheduling. (In Python 3, reproducibility between interpreter launches also requires\n",
       "    use of the `PYTHONHASHSEED` environment variable to control hash randomization).\n",
       "min_count : int\n",
       "    Ignores all words with total frequency lower than this.\n",
       "max_vocab_size : int\n",
       "    Limits the RAM during vocabulary building; if there are more unique\n",
       "    words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM.\n",
       "    Set to `None` for no limit.\n",
       "sample : float\n",
       "    The threshold for configuring which higher-frequency words are randomly downsampled,\n",
       "    useful range is (0, 1e-5).\n",
       "workers : int\n",
       "    Use these many worker threads to train the model (=faster training with multicore machines).\n",
       "iter : int\n",
       "    Number of iterations (epochs) over the corpus.\n",
       "hs : int {1,0}\n",
       "    If 1, hierarchical softmax will be used for model training.\n",
       "    If set to 0, and `negative` is non-zero, negative sampling will be used.\n",
       "negative : int\n",
       "    If > 0, negative sampling will be used, the int for negative specifies how many \"noise words\"\n",
       "    should be drawn (usually between 5-20).\n",
       "    If set to 0, no negative sampling is used.\n",
       "dm_mean : int {1,0}\n",
       "    If 0 , use the sum of the context word vectors. If 1, use the mean.\n",
       "    Only applies when `dm` is used in non-concatenative mode.\n",
       "dm_concat : int {1,0}\n",
       "    If 1, use concatenation of context vectors rather than sum/average;\n",
       "    Note concatenation results in a much-larger model, as the input\n",
       "    is no longer the size of one (sampled or arithmetically combined) word vector, but the\n",
       "    size of the tag(s) and all words in the context strung together.\n",
       "dm_tag_count : int\n",
       "    Expected constant number of document tags per document, when using\n",
       "    dm_concat mode; default is 1.\n",
       "dbow_words : int {1,0}\n",
       "    If set to 1 trains word-vectors (in skip-gram fashion) simultaneous with DBOW\n",
       "    doc-vector training; If 0, only trains doc-vectors (faster).\n",
       "trim_rule : function\n",
       "    Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary,\n",
       "    be trimmed away, or handled using the default (discard if word count < min_count).\n",
       "    Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`),\n",
       "    or a callable that accepts parameters (word, count, min_count) and returns either\n",
       "    :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`.\n",
       "    Note: The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part\n",
       "    of the model.\n",
       "callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`\n",
       "    List of callbacks that need to be executed/run at specific stages during training.\n",
       "\u001b[0;31mFile:\u001b[0m           ~/anaconda3/envs/jgi-data-week-workshop/lib/python3.6/site-packages/gensim/models/doc2vec.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This is how you can check the API usage.\n",
    "# You can do this in the notebook kernel session or \n",
    "# in a jupyter lab console session associated to this kernel\n",
    "\n",
    "?Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assess accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check of our model fit,\n",
    "we take the first 1000 abstract from the entire train corpus,\n",
    "and see whether for the given text, the most similar abstract is itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_n = 1000\n",
    "sample_corpus = train_corpus[:sample_n]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranks = []\n",
    "for doc_id in range(len(sample_corpus)):\n",
    "    # The inferred_vector of this document\n",
    "    inferred_vector = d2v_model.infer_vector(sample_corpus[doc_id].words)\n",
    "    # Get the most similar document rankings across entire `train_corpus`\n",
    "    # in the form of\n",
    "    # [(499, 0.7327660322189331),\n",
    "    #  (8835, 0.6161227822303772),\n",
    "    #  (981, 0.5684570074081421),\n",
    "    #  ...]\n",
    "    sims = d2v_model.docvecs.most_similar([inferred_vector], \n",
    "                                          topn=len(d2v_model.docvecs))\n",
    "    # The index position for `doc_id`.\n",
    "    # If this abstract is most similar to itself, rank should be 0\n",
    "    rank = [docid for docid, sim in sims].index(doc_id)\n",
    "    \n",
    "    ranks.append(rank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({0: 996, 1: 2, 2: 1, 3: 1})\n",
      "Accuracy: 99.6%\n"
     ]
    }
   ],
   "source": [
    "print(collections.Counter(ranks))\n",
    "\n",
    "accuracy = np.sum(np.array(ranks) == 0) / sample_n * 100\n",
    "print(f\"Accuracy: {accuracy}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommendation engine based on abstract embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose documents from `test_corpus` are from authors that are interested in finding collaborators for future works.\n",
    "\n",
    "For a small sample of `test_corpus` documents, we test to find the most similar documents from `train_corpus`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TaggedDocument(Adaptation and validation of antibody-ELISA using dried blood spots on filter paper for epidemiological surveys of tsetse-transmitted trypanosomosis in cattle. The indirect enzyme-linked immunosorbent assay (ELISA) for the detection of anti-trypanosomal antibodies in bovine serum was adapted for use with dried blood spots on filter paper. Absorbance (450 nm) results for samples were expressed as percent positivity, i.e. percentage of the median absorbance result of four replicates of the strong positive control serum. The antibody-ELISA was evaluated in Zambia for use in epidemiological surveys of the prevalence of tsetse-transmitted bovine trypanosomosis. Known negative samples (sera, n = 209; blood spots, n = 466) were obtained from cattle from closed herds in tsetse-free areas close to Lusaka. Known positive samples (sera, n = 367; blood spots, n = 278) were obtained from cattle in Zambia's Central, Lusaka and Eastern Provinces, diagnosed as being infected with Trypanosoma brucei, T. congolense, or T. vivax using the phase-contrast buffy-coat technique or Giemsa-stained thick and thin blood smears. For sera (at a cut-off value of 23.0% positivity) sensitivity and specificity were 86.1 and 95.2%, respectively. For bloodspots (at a cut-off value of 18.8% positivity) sensitivity and specificity were 96.8 and 95.7%, respectively. The implications of persistence of antibodies following treatment or self-cure are discussed., [0]) \n",
      "\n",
      "TaggedDocument(How human brucellosis incidence in urban Kampala can be reduced most efficiently? A stochastic risk assessment of informally-marketed milk. In Kampala, Uganda, studies have shown a significant incidence of human brucellosis. A stochastic risk assessment involving two field surveys (cattle farms and milk shops) and a medical record survey was conducted to assess the risk of human brucellosis infection through consumption of informally marketed raw milk potentially infected with Brucella abortus in Kampala and to identify the best control options., [1]) \n",
      "\n",
      "TaggedDocument(Herd prevalence of bovine brucellosis and analysis of risk factors in cattle in urban and peri-urban areas of the Kampala economic zone, Uganda. Human brucellosis has been found to be prevalent in the urban areas of Kampala, the capital city of Uganda. A cross-sectional study was designed to generate precise information on the prevalence of brucellosis in cattle and risk factors for the disease in its urban and peri-urban dairy farming systems., [2]) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for doc_id in range(3):\n",
    "    print(test_corpus[doc_id], \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_similar_n = 3\n",
    "similar = []\n",
    "\n",
    "for test_doc_id in range(len(test_corpus)):\n",
    "    words = test_corpus[test_doc_id].words\n",
    "    inferred_vector = d2v_model.infer_vector(words, steps=20)\n",
    "    top_sim = d2v_model.docvecs.most_similar(\n",
    "        [inferred_vector], topn = top_similar_n)\n",
    "    similar.append(top_sim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# rank test abstract by similar scores\n",
    "most_similar = []\n",
    "for test_doc_id in range(len(test_corpus)):\n",
    "    item = {\n",
    "        \"test_doc_id\": test_doc_id,\n",
    "        \"train_doc_id\": similar[test_doc_id][0][0],\n",
    "        \"score\": similar[test_doc_id][0][1],\n",
    "    }\n",
    "    most_similar.append(item)\n",
    "    \n",
    "most_similar = pd.DataFrame(most_similar).sort_values(by=\"score\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>test_doc_id</th>\n",
       "      <th>train_doc_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>0.729217</td>\n",
       "      <td>253</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>0.712785</td>\n",
       "      <td>368</td>\n",
       "      <td>5598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>401</th>\n",
       "      <td>0.707592</td>\n",
       "      <td>401</td>\n",
       "      <td>5598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>0.682189</td>\n",
       "      <td>293</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0.673601</td>\n",
       "      <td>138</td>\n",
       "      <td>4084</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        score  test_doc_id  train_doc_id\n",
       "253  0.729217          253          8185\n",
       "368  0.712785          368          5598\n",
       "401  0.707592          401          5598\n",
       "293  0.682189          293          8185\n",
       "138  0.673601          138          4084"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's select top 2 test abstracts that have the best match from `train_corpus`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Abstract 253:\n",
      "content: Rho proteins and cancer. The Rho family of GTPases has been intensively studied for their roles in signal transduction processes leading to cytoskeletal-dependent responses, including cell migration and phagocytosis. In addition, they are important regulators of cell cycle progression and affect the expression of a number of genes, including those for matrix-degrading proteases implicated in cancer invasion. So far, the expression of some Rho family members has been found to be increased in some human cancers, and some cancer-associated mutations in Rho family regulators have been characterized. This makes Rho protein signalling pathways attractive targets for cancer therapy. However, there is little evidence so far from animal studies to define if and how Rho proteins contribute to cancer cell proliferation, survival, invasion and metastasis.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 0: id 8185, similarity 0.7292169332504272\n",
      "\t Regulation of endocytic traffic by Rho GTPases. The members of the Rho subfamily of small GTPases are key regulators of the actin cytoskeleton. However, recent studies have provided evidence for multiple additional roles for these signalling proteins in controlling endocytic traffic. Here we review our current understanding of Rho GTPase action within the endocytic pathway and examine the potential points of convergence with the more established, actin-based functions of these signalling proteins.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 1: id 5598, similarity 0.6352338790893555\n",
      "\t Ran-GTP coordinates regulation of microtubule nucleation and dynamics during mitotic-spindle assembly. It was recently reported that GTP-bound Ran induces microtubule and pseudo-spindle assembly in mitotic egg extracts in the absence of chromosomes and centrosomes, and that chromosomes induce the assembly of spindle microtubules in these extracts through generation of Ran-GTP. Here we examine the effects of Ran-GTP on microtubule nucleation and dynamics and show that Ran-GTP has independent effects on both the nucleation activity of centrosomes and the stability of centrosomal microtubules. We also show that inhibition of Ran-GTP production, even in the presence of duplicated centrosomes and kinetochores, prevents assembly of a bipolar spindle in M-phase extracts.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 2: id 4337, similarity 0.5987986326217651\n",
      "\t The mechanism underlying inhibition of saccadic return. Human observers take longer to re-direct gaze to a previously fixated location. Although there has been some exploration of the characteristics of inhibition of saccadic return (ISR), the exact mechanisms by which ISR operates are currently unknown. In the framework of accumulation models of response times, in which evidence is integrated over time to a response threshold, ISR could reflect a reduction in the rate of accumulation for saccades to return locations or an increase in the effective criterion for response. In two experiments, participants generated sequences of three saccades, in response to a peripheral or a central cue. ISR occurred across these manipulations: saccade latency was consistently increased for movements to the immediately previously fixated location. Latency distributions from individual observers were fit with a Linear Ballistic Accumulator model. ISR was best accounted for as a change in the accumulation rate. We suggest this parameter represents the overall desirability of a particular course of action, the evidence for which may be derived from a variety of sensory and non-sensory sources.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "# Abstract 368:\n",
      "content: Regulating Rho GTPases and their regulators. Rho GTPases regulate cytoskeletal and cell adhesion dynamics and thereby coordinate a wide range of cellular processes, including cell migration, cell polarity and cell cycle progression. Most Rho GTPases cycle between a GTP-bound active conformation and a GDP-bound inactive conformation to regulate their ability to activate effector proteins and to elicit cellular responses. However, it has become apparent that Rho GTPases are regulated by post-translational modifications and the formation of specific protein complexes, in addition to GTP-GDP cycling. The canonical regulators of Rho GTPases - guanine nucleotide exchange factors, GTPase-activating proteins and guanine nucleotide dissociation inhibitors - are regulated similarly, creating a complex network of interactions to determine the precise spatiotemporal activation of Rho GTPases.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 0: id 5598, similarity 0.7127853631973267\n",
      "\t Ran-GTP coordinates regulation of microtubule nucleation and dynamics during mitotic-spindle assembly. It was recently reported that GTP-bound Ran induces microtubule and pseudo-spindle assembly in mitotic egg extracts in the absence of chromosomes and centrosomes, and that chromosomes induce the assembly of spindle microtubules in these extracts through generation of Ran-GTP. Here we examine the effects of Ran-GTP on microtubule nucleation and dynamics and show that Ran-GTP has independent effects on both the nucleation activity of centrosomes and the stability of centrosomal microtubules. We also show that inhibition of Ran-GTP production, even in the presence of duplicated centrosomes and kinetochores, prevents assembly of a bipolar spindle in M-phase extracts.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 1: id 3638, similarity 0.636596143245697\n",
      "\t Fourteen-year final report of the randomized PDRG-UK trial comparing three initial treatments in PD. Ten-year follow-up results from the Parkinson's Disease Research Group of the United Kingdom trial demonstrated that there were no long-term advantages to initiating treatment with bromocriptine compared with l-dopa in early Parkinson disease (PD). Increased mortality in patients on selegiline combined with l-dopa led to premature termination of this arm after 6 years.\n",
      "\n",
      "\n",
      "\t ## Matched abstract 2: id 5369, similarity 0.6260905265808105\n",
      "\t Regulation of small GTP-binding proteins by insulin. Several members of the extensive family of small GTP-binding proteins are regulated by insulin, and have been implicated in insulin action on glucose uptake. These proteins are themselves negatively regulated by a series of specific GAPs (GTPase-activating proteins). Interestingly, there is increasing evidence to suggest that PKB (protein kinase B)-dependent phosphorylation of some GAPs may relieve this negative regulation and so lead to the activation of the target small GTP-binding protein. We review recent evidence that this may be the case, and place specific emphasis on the role of these pathways in insulin-stimulated glucose uptake.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "top_n = 2\n",
    "preview_n = 3\n",
    "\n",
    "for test_doc_id in most_similar.test_doc_id[:top_n]:\n",
    "    print(f\"# Abstract {test_doc_id}:\")\n",
    "    words = test_corpus[test_doc_id].words\n",
    "    print(f\"content: {words}\")\n",
    "    print(\"\\n\")\n",
    "    \n",
    "    for i in range(preview_n):\n",
    "        train_doc_id = similar[test_doc_id][i][0]\n",
    "        similarity_score = similar[test_doc_id][i][1]\n",
    "        print(f\"\\t ## Matched abstract {i}: id {train_doc_id}, similarity {similarity_score}\")\n",
    "        words = train_corpus[train_doc_id].words\n",
    "        print(f\"\\t {words}\")\n",
    "        print(\"\\n\")\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feeling lucky with some random text?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. We set out a generalized linear model framework for the synthesis of data from randomized controlled trials. A common model is described, taking the form of a linear regression for both fixed and random effects synthesis, which can be implemented with normal, binomial, Poisson, and multinomial data. The familiar logistic model for meta-analysis with binomial data is a generalized linear model with a logit link function, which is appropriate for probability outcomes. The same linear regression framework can be applied to continuous outcomes, rate models, competing risks, or ordered category outcomes by using other link functions, such as identity, log, complementary log-log, and probit link functions. The common core model for the linear predictor can be applied to pairwise meta-analysis, indirect comparisons, synthesis of multiarm trials, and mixed treatment comparisons, also known as network meta-analysis, without distinction. We take a Bayesian approach to estimation and provide WinBUGS program code for a Bayesian analysis using Markov chain Monte Carlo simulation. An advantage of this approach is that it is straightforward to extend to shared parameter models where different randomized controlled trials report outcomes in different formats but from a common underlying model. Use of the generalized linear model framework allows us to present a unified account of how models can be compared using the deviance information criterion and how goodness of fit can be assessed using the residual deviance. The approach is illustrated through a range of worked examples for commonly encountered evidence formats.',\n",
       " 0.5261853933334351)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Searching for the causal effects of body mass index in over 300 000 participants in UK Biobank, using Mendelian randomization.\"\n",
    "\n",
    "def feeling_lucky(text, model=d2v_model, train_corpus=train_corpus):\n",
    "    inferred_vector = model.infer_vector(text, steps=20)\n",
    "    top_sim = model.docvecs.most_similar(\n",
    "        [inferred_vector], topn = 1)\n",
    "    matched_abstract = train_corpus[top_sim[0][0]].words\n",
    "    similarity_score = top_sim[0][1]\n",
    "    return matched_abstract, similarity_score\n",
    "\n",
    "feeling_lucky(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this is a very crude demo (small training set and lack of pre-processing and model tuning).\n",
    "\n",
    "Below is the overall distribution of top similarity scores. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>test_doc_id</th>\n",
       "      <th>train_doc_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>0.687133</td>\n",
       "      <td>256</td>\n",
       "      <td>7012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>0.681590</td>\n",
       "      <td>154</td>\n",
       "      <td>6335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0.679621</td>\n",
       "      <td>138</td>\n",
       "      <td>4084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>0.678780</td>\n",
       "      <td>260</td>\n",
       "      <td>2168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>0.676889</td>\n",
       "      <td>300</td>\n",
       "      <td>8202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>403</th>\n",
       "      <td>0.671718</td>\n",
       "      <td>403</td>\n",
       "      <td>5784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>0.665003</td>\n",
       "      <td>293</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>0.661996</td>\n",
       "      <td>368</td>\n",
       "      <td>5598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>440</th>\n",
       "      <td>0.656170</td>\n",
       "      <td>440</td>\n",
       "      <td>1988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286</th>\n",
       "      <td>0.653303</td>\n",
       "      <td>286</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>0.652377</td>\n",
       "      <td>315</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>0.652225</td>\n",
       "      <td>389</td>\n",
       "      <td>4430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>407</th>\n",
       "      <td>0.642343</td>\n",
       "      <td>407</td>\n",
       "      <td>5878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>354</th>\n",
       "      <td>0.641203</td>\n",
       "      <td>354</td>\n",
       "      <td>5598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310</th>\n",
       "      <td>0.638505</td>\n",
       "      <td>310</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>360</th>\n",
       "      <td>0.636913</td>\n",
       "      <td>360</td>\n",
       "      <td>4872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>401</th>\n",
       "      <td>0.635799</td>\n",
       "      <td>401</td>\n",
       "      <td>5598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>0.628839</td>\n",
       "      <td>103</td>\n",
       "      <td>2687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350</th>\n",
       "      <td>0.625694</td>\n",
       "      <td>350</td>\n",
       "      <td>5370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>0.624060</td>\n",
       "      <td>272</td>\n",
       "      <td>5743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>0.620159</td>\n",
       "      <td>282</td>\n",
       "      <td>7352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>0.619087</td>\n",
       "      <td>198</td>\n",
       "      <td>358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>423</th>\n",
       "      <td>0.618067</td>\n",
       "      <td>423</td>\n",
       "      <td>6234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>0.614517</td>\n",
       "      <td>357</td>\n",
       "      <td>5784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>0.613666</td>\n",
       "      <td>247</td>\n",
       "      <td>5250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>0.612984</td>\n",
       "      <td>413</td>\n",
       "      <td>8185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>0.612803</td>\n",
       "      <td>153</td>\n",
       "      <td>5161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>0.612589</td>\n",
       "      <td>344</td>\n",
       "      <td>7012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>0.611871</td>\n",
       "      <td>246</td>\n",
       "      <td>5118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>0.611774</td>\n",
       "      <td>82</td>\n",
       "      <td>3665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.444359</td>\n",
       "      <td>9</td>\n",
       "      <td>3852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325</th>\n",
       "      <td>0.441462</td>\n",
       "      <td>325</td>\n",
       "      <td>6890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>0.441223</td>\n",
       "      <td>66</td>\n",
       "      <td>4844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>430</th>\n",
       "      <td>0.441026</td>\n",
       "      <td>430</td>\n",
       "      <td>227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0.440331</td>\n",
       "      <td>86</td>\n",
       "      <td>727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212</th>\n",
       "      <td>0.439796</td>\n",
       "      <td>212</td>\n",
       "      <td>8219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>0.437693</td>\n",
       "      <td>313</td>\n",
       "      <td>338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.435546</td>\n",
       "      <td>12</td>\n",
       "      <td>7238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>324</th>\n",
       "      <td>0.433955</td>\n",
       "      <td>324</td>\n",
       "      <td>7064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>363</th>\n",
       "      <td>0.432634</td>\n",
       "      <td>363</td>\n",
       "      <td>4739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>0.432582</td>\n",
       "      <td>209</td>\n",
       "      <td>1533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>0.432152</td>\n",
       "      <td>119</td>\n",
       "      <td>1951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>0.432091</td>\n",
       "      <td>177</td>\n",
       "      <td>1155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>0.430848</td>\n",
       "      <td>167</td>\n",
       "      <td>2945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.429054</td>\n",
       "      <td>4</td>\n",
       "      <td>396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>0.428183</td>\n",
       "      <td>102</td>\n",
       "      <td>773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>420</th>\n",
       "      <td>0.427085</td>\n",
       "      <td>420</td>\n",
       "      <td>8158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>0.426860</td>\n",
       "      <td>419</td>\n",
       "      <td>2423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.423371</td>\n",
       "      <td>19</td>\n",
       "      <td>5037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>0.422620</td>\n",
       "      <td>380</td>\n",
       "      <td>6187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>0.421899</td>\n",
       "      <td>108</td>\n",
       "      <td>3096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>0.421886</td>\n",
       "      <td>298</td>\n",
       "      <td>6961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>0.421874</td>\n",
       "      <td>250</td>\n",
       "      <td>2894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>0.420970</td>\n",
       "      <td>84</td>\n",
       "      <td>3125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>0.418832</td>\n",
       "      <td>230</td>\n",
       "      <td>1464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.412839</td>\n",
       "      <td>0</td>\n",
       "      <td>425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>432</th>\n",
       "      <td>0.405065</td>\n",
       "      <td>432</td>\n",
       "      <td>1235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>0.403615</td>\n",
       "      <td>224</td>\n",
       "      <td>5862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>0.402851</td>\n",
       "      <td>348</td>\n",
       "      <td>261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437</th>\n",
       "      <td>0.387544</td>\n",
       "      <td>437</td>\n",
       "      <td>5425</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>441 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        score  test_doc_id  train_doc_id\n",
       "256  0.687133          256          7012\n",
       "154  0.681590          154          6335\n",
       "138  0.679621          138          4084\n",
       "260  0.678780          260          2168\n",
       "300  0.676889          300          8202\n",
       "403  0.671718          403          5784\n",
       "293  0.665003          293          8185\n",
       "368  0.661996          368          5598\n",
       "440  0.656170          440          1988\n",
       "286  0.653303          286          8185\n",
       "315  0.652377          315          8185\n",
       "389  0.652225          389          4430\n",
       "407  0.642343          407          5878\n",
       "354  0.641203          354          5598\n",
       "310  0.638505          310          8185\n",
       "360  0.636913          360          4872\n",
       "401  0.635799          401          5598\n",
       "103  0.628839          103          2687\n",
       "350  0.625694          350          5370\n",
       "272  0.624060          272          5743\n",
       "282  0.620159          282          7352\n",
       "198  0.619087          198           358\n",
       "423  0.618067          423          6234\n",
       "357  0.614517          357          5784\n",
       "247  0.613666          247          5250\n",
       "413  0.612984          413          8185\n",
       "153  0.612803          153          5161\n",
       "344  0.612589          344          7012\n",
       "246  0.611871          246          5118\n",
       "82   0.611774           82          3665\n",
       "..        ...          ...           ...\n",
       "9    0.444359            9          3852\n",
       "325  0.441462          325          6890\n",
       "66   0.441223           66          4844\n",
       "430  0.441026          430           227\n",
       "86   0.440331           86           727\n",
       "212  0.439796          212          8219\n",
       "313  0.437693          313           338\n",
       "12   0.435546           12          7238\n",
       "324  0.433955          324          7064\n",
       "363  0.432634          363          4739\n",
       "209  0.432582          209          1533\n",
       "119  0.432152          119          1951\n",
       "177  0.432091          177          1155\n",
       "167  0.430848          167          2945\n",
       "4    0.429054            4           396\n",
       "102  0.428183          102           773\n",
       "420  0.427085          420          8158\n",
       "419  0.426860          419          2423\n",
       "19   0.423371           19          5037\n",
       "380  0.422620          380          6187\n",
       "108  0.421899          108          3096\n",
       "298  0.421886          298          6961\n",
       "250  0.421874          250          2894\n",
       "84   0.420970           84          3125\n",
       "230  0.418832          230          1464\n",
       "0    0.412839            0           425\n",
       "432  0.405065          432          1235\n",
       "224  0.403615          224          5862\n",
       "348  0.402851          348           261\n",
       "437  0.387544          437          5425\n",
       "\n",
       "[441 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [introduction to doc2vec](https://blog.acolyer.org/2016/06/01/distributed-representations-of-sentences-and-documents/)\n",
    "- [sentiment analysis](http://linanqiu.github.io/2015/10/07/word2vec-sentiment/)\n",
    "- https://medium.com/@ermolushka/text-clusterization-using-python-and-doc2vec-8c499668fa61\n",
    "- https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-lee.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc-autonumbering": false,
  "toc-showmarkdowntxt": true
 },
 "nbformat": 4,
 "nbformat_minor": 2
}