notebooks/code2vec BOW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# code2vec using bag-of-words (BOW) approaches\n",
    "\n",
    "###  As a baseline for learning representations\n",
    "\n",
    "VECTORIZE - Create code vectors using different strategies:\n",
    "\n",
    "1. Word Vectors\n",
    "    * Count\n",
    "    * TF-IDF\n",
    "    * Frequency (ratio)\n",
    "    * Binary (Presence / Absence)\n",
    "2. Token Vectors\n",
    "    * Count\n",
    "3. AST Vectors\n",
    "\n",
    "TRAIN - Learn representations using these combinations of code vectors for a machine to differentiate between:\n",
    "\n",
    "- correct code (code submission pass the testcases)\n",
    "- wrong code (code submission fail the testcases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "font = {'weight': 'bold', 'size': 20}\n",
    "plt.rc('font', **font)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VECTORIZE\n",
    "\n",
    "**Programming data**: Student develop programs locally for the laboratory sheets of computer programming courses at our university. These programs are submitted by them to an automatic grading platform that runs some test cases specified by the lecturer on each program. A JSON output for those testcases and whether the program passed them or not along with the actual code is stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/raw/programming_data.json\r\n"
     ]
    }
   ],
   "source": [
    "!ls data/raw/programming_data.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataframe = pd.read_json('data/raw/programming_data.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>academic_year_0</th>\n",
       "      <th>academic_year_1</th>\n",
       "      <th>correct</th>\n",
       "      <th>date</th>\n",
       "      <th>extension</th>\n",
       "      <th>ip</th>\n",
       "      <th>module</th>\n",
       "      <th>task</th>\n",
       "      <th>upload</th>\n",
       "      <th>user</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016</td>\n",
       "      <td>2017</td>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-19 14:11:41</td>\n",
       "      <td>py</td>\n",
       "      <td></td>\n",
       "      <td>ca277</td>\n",
       "      <td>add.py</td>\n",
       "      <td>#!/usr/bin/env python\\n\\na = int(raw_input())\\...</td>\n",
       "      <td>b9e7e608-6036-4d44-8770-a7036176b53c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016</td>\n",
       "      <td>2017</td>\n",
       "      <td>True</td>\n",
       "      <td>2016-09-19 14:17:33</td>\n",
       "      <td>py</td>\n",
       "      <td></td>\n",
       "      <td>ca277</td>\n",
       "      <td>concat-lines.py</td>\n",
       "      <td>#!/usr/bin/env python\\n\\na = str(raw_input())\\...</td>\n",
       "      <td>b9e7e608-6036-4d44-8770-a7036176b53c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  academic_year_0 academic_year_1  correct                date extension ip  \\\n",
       "0            2016            2017     True 2016-09-19 14:11:41        py      \n",
       "1            2016            2017     True 2016-09-19 14:17:33        py      \n",
       "\n",
       "  module             task                                             upload  \\\n",
       "0  ca277           add.py  #!/usr/bin/env python\\n\\na = int(raw_input())\\...   \n",
       "1  ca277  concat-lines.py  #!/usr/bin/env python\\n\\na = str(raw_input())\\...   \n",
       "\n",
       "                                   user  \n",
       "0  b9e7e608-6036-4d44-8770-a7036176b53c  \n",
       "1  b9e7e608-6036-4d44-8770-a7036176b53c  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'591,707'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:,}'.format(len(dataframe))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab only code submissions from Python modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PYTHON_MODULES = [\n",
    "    'ca116', \n",
    "    'ca117', \n",
    "    'ca177', \n",
    "    'ca277', \n",
    "    'ca278',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataframe = dataframe[dataframe['module'].isin(PYTHON_MODULES)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'490,820'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:,}'.format(len(dataframe))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Target value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    296369\n",
       "True     194451\n",
       "Name: correct, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe.correct.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove comments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "def remove_comments(text):\n",
    "    return re.sub(re.compile('#.*?\\n'), '', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab docs (code submissions) and labels (correct or not):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_docs_and_labels(df):\n",
    "    _docs = []\n",
    "    _labels = []\n",
    "    for index in df.index:\n",
    "        # Program\n",
    "        code = remove_comments(\n",
    "            df.at[index, 'upload']\n",
    "        )\n",
    "        _docs.append(code)\n",
    "        # Label\n",
    "        label = int(df.at[index, 'correct'])\n",
    "        _labels.append(label)\n",
    "    return _docs, _labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "docs, labels = get_docs_and_labels(dataframe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'490,820'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:,}'.format(len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'\\na = int(raw_input())\\nb = int(raw_input())\\n\\nprint a + b\\n\\n\\n'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Programs as word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "NUM_WORDS = 2000 # Originally 231,659 words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t = Tokenizer(num_words=NUM_WORDS, \n",
    "              filters='\\t\\n', \n",
    "              lower=True, \n",
    "              split=' ', \n",
    "              char_level=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t.fit_on_texts(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "552539"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# word_counts: a dictionary of words and their counts.\n",
    "t.word_counts['if'] # word count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ordered_words = sorted(t.word_counts.iteritems(), key=lambda (k,v): (v,k), reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=: 2440154\n",
      "i: 910221\n",
      "+: 575607\n",
      "if: 552539\n",
      "def: 522536\n"
     ]
    }
   ],
   "source": [
    "N = 5\n",
    "for key, value in ordered_words[:N]:\n",
    "    print \"%s: %s\" % (key, value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number docs: 490,820'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# document_count: an integer count of the total number of documents that were used to fit the Tokenizer.\n",
    "'Number docs: {:,}'.format(t.document_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# word_index: a dictionary of words and their uniquely assigned integers.\n",
    "t.word_index['if'] # index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "298487"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# word_docs: a dictionary of words and how many documents each appeared in.\n",
    "t.word_docs['if']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i) Count: count of each word in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_docs_count = t.texts_to_matrix(docs, mode='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  2.,  0., ...,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_docs_count[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(490820, 2000)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_docs_count.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filename = 'data/processed/word_count'\n",
    "np.save(filename, encoded_docs_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/word_labels', labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ii) TF-IDF: Text Frequency-Inverse DocumentFrequency (TF-IDF) scoring for each word in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded_docs_tfidf = t.texts_to_matrix(docs, mode='tfidf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.        ,  1.25768178,  0.        , ...,  0.        ,\n",
       "        0.        ,  0.        ])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_docs_tfidf[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/word_tfidf', encoded_docs_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iii) Frequency: frequency of each word as a ratio of words within each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_docs_freq = t.texts_to_matrix(docs, mode='freq')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0. ,  0.2,  0. , ...,  0. ,  0. ,  0. ])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_docs_freq[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/word_freq', encoded_docs_freq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iv) Binary: Whether or not each word is present in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_docs_binary = t.texts_to_matrix(docs, mode='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  1.,  0., ...,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_docs_binary[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save('data/processed/word_binary', encoded_docs_binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Programs as tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tokenize import generate_tokens\n",
    "from StringIO import StringIO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate token IDs and token words. Token IDs are more general than token words (see the \"Program Vectors\" notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sample_code = '''print(\"Hello World!\")'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'print'), (51, '('), (3, '\"Hello World!\"'), (51, ')'), (0, '')]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(t[0], t[1]) for t in list(generate_tokens(StringIO(sample_code).readline))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "token_ids = []\n",
    "token_words = []\n",
    "token_labels = []\n",
    "i = 0\n",
    "while i < len(docs):\n",
    "    doc = docs[i]\n",
    "    label = labels[i]\n",
    "    try:\n",
    "        tokens = [(t[0], t[1]) for t in list(generate_tokens(StringIO(doc).readline))]\n",
    "        # Token ID\n",
    "        token_ids.append(\n",
    "            [token[0] for token in tokens]\n",
    "        )\n",
    "        # Token\n",
    "        token_words.append(\n",
    "            [token[1] for token in tokens]\n",
    "        )\n",
    "        token_labels.append(\n",
    "            label\n",
    "        )\n",
    "    except:\n",
    "        pass\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Token categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 51, 3, 51, 0]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[t[0] for t in list(generate_tokens(StringIO(sample_code).readline))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[54,\n",
       " 1,\n",
       " 51,\n",
       " 1,\n",
       " 51,\n",
       " 1,\n",
       " 51,\n",
       " 51,\n",
       " 51,\n",
       " 4,\n",
       " 1,\n",
       " 51,\n",
       " 1,\n",
       " 51,\n",
       " 1,\n",
       " 51,\n",
       " 51,\n",
       " 51,\n",
       " 4,\n",
       " 54,\n",
       " 1,\n",
       " 1,\n",
       " 51,\n",
       " 1,\n",
       " 4,\n",
       " 54,\n",
       " 54,\n",
       " 0]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number encoded docs: 472,087'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Number encoded docs: {:,}'.format(len(token_ids))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "token_cat_docs = [\n",
    "    ' '.join([str(item) for item in array]) for array in token_ids\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'54 1 51 1 51 1 51 51 51 4 1 51 1 51 1 51 51 51 4 54 1 1 51 1 4 54 54 0'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_cat_docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number token category docs: 472,087'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Number token category docs: {:,}'.format(len(token_cat_docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_t = Tokenizer(num_words=NUM_WORDS,\n",
    "                  split=' ',\n",
    "                  char_level=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_t.fit_on_texts(token_cat_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51: 20368593\n",
      "1: 18075194\n",
      "4: 5886806\n",
      "2: 2317086\n",
      "54: 1996531\n"
     ]
    }
   ],
   "source": [
    "ordered_categories = sorted(cat_t.word_counts.iteritems(), key=lambda (k,v): (v,k), reverse=True)\n",
    "N = 5\n",
    "for key, value in ordered_categories[:N]:\n",
    "    print \"%s: %s\" % (key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i) Count: count of each token word in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_category_tokens_count = cat_t.texts_to_matrix(token_cat_docs, mode='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  0.,  11.,   9., ...,   0.,   0.,   0.])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_category_tokens_count[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/token_cat_count', encoded_category_tokens_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/token_labels', token_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Token words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['print', '(', '\"Hello World!\"', ')', '']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[t[1] for t in list(generate_tokens(StringIO(sample_code).readline))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'\\n',\n",
       " u'a',\n",
       " u'=',\n",
       " u'int',\n",
       " u'(',\n",
       " u'raw_input',\n",
       " u'(',\n",
       " u')',\n",
       " u')',\n",
       " u'\\n',\n",
       " u'b',\n",
       " u'=',\n",
       " u'int',\n",
       " u'(',\n",
       " u'raw_input',\n",
       " u'(',\n",
       " u')',\n",
       " u')',\n",
       " u'\\n',\n",
       " u'\\n',\n",
       " u'print',\n",
       " u'a',\n",
       " u'+',\n",
       " u'b',\n",
       " u'\\n',\n",
       " u'\\n',\n",
       " u'\\n',\n",
       " '']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_words[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number encoded docs: 472,087'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Number encoded docs: {:,}'.format(len(token_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "token_docs = [\n",
    "    ' '.join(array) for array in token_words\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'\\n a = int ( raw_input ( ) ) \\n b = int ( raw_input ( ) ) \\n \\n print a + b \\n \\n \\n '"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number token docs: 472,087'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Number token docs: {:,}'.format(len(token_docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "token_t = Tokenizer(num_words=NUM_WORDS, \n",
    "                    filters='\\t\\n', \n",
    "                    lower=False, \n",
    "                    split=' ', \n",
    "                    char_level=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "token_t.fit_on_texts(token_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "): 3556931\n",
      "(: 3556907\n",
      "=: 2581991\n",
      ":: 2248901\n",
      ".: 2011442\n"
     ]
    }
   ],
   "source": [
    "ordered_tokens = sorted(token_t.word_counts.iteritems(), key=lambda (k,v): (v,k), reverse=True)\n",
    "N = 5\n",
    "for key, value in ordered_tokens[:N]:\n",
    "    print \"%s: %s\" % (key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i) Count: count of each token word in the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_word_tokens_count = token_t.texts_to_matrix(token_docs, mode='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  4.,  4., ...,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_word_tokens_count[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/token_word_count', encoded_word_tokens_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Programs as Abstract Syntax Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract a representation of the AST by using a BFS approach:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import ast\n",
    "\n",
    "def _strip_docstring(body):\n",
    "    first = body[0]\n",
    "    if isinstance(first, ast.Expr) and isinstance(first.value, ast.Str):\n",
    "        return body[1:]\n",
    "    return body\n",
    "\n",
    "def get_ast_repr(node):\n",
    "    \n",
    "    visited = set()\n",
    "    queue = [ [node, None, None, False] ]\n",
    "    output = []\n",
    "    \n",
    "    while queue:\n",
    "        \n",
    "        vertex, value, name, end = queue.pop(0)\n",
    "        \n",
    "        # OUTPUT\n",
    "        output.append(get_leaf(vertex, value, name, end))\n",
    "        \n",
    "        if vertex not in visited:\n",
    "            \n",
    "            visited.add(vertex)\n",
    "            \n",
    "            if hasattr(vertex, '_fields'):\n",
    "                \n",
    "                for field_name, field_value in zip(vertex._fields, \n",
    "                                               (getattr(vertex, attr) for attr in vertex._fields)):\n",
    "\n",
    "                    if isinstance(field_value, ast.AST):\n",
    "                        queue.append([field_value, field_name, vertex, False])\n",
    "                    \n",
    "                    elif isinstance(field_value, list):\n",
    "\n",
    "                        if field_name == 'body':\n",
    "                            field_name = _strip_docstring(field_value)\n",
    "                        for item in field_value:\n",
    "                            if isinstance(item, ast.AST):\n",
    "                                queue.append([item, field_name, vertex, False])\n",
    "                            else:\n",
    "                                queue.append([item, field_name, vertex, True])\n",
    "\n",
    "                    else:\n",
    "                        queue.append((field_value, field_name, vertex, True))\n",
    "                   \n",
    "    return output\n",
    "\n",
    "def get_leaf(node, value, parent, end):\n",
    "    \n",
    "    node_name = node.__class__.__name__\n",
    "    node_name = next(node_name) if node_name is None else node_name\n",
    "    if node_name == 'str': node_name = str(node)\n",
    "\n",
    "    return {\n",
    "        'Node': node, \n",
    "        'Parent': parent, \n",
    "        'Name': node_name,\n",
    "        'Value': value,\n",
    "        'End': end,\n",
    "    }\n",
    "\n",
    "def get_ast_pairs(tree):\n",
    "    names = {}\n",
    "    tuples = []\n",
    "    for leaf in tree:\n",
    "        # Values\n",
    "        node = leaf['Node']\n",
    "        node_name = leaf['Name']\n",
    "        parent = leaf['Parent']\n",
    "        value = leaf['Value']\n",
    "        # Save name\n",
    "        names[node] = node_name\n",
    "        if node is None or parent is None:\n",
    "            continue\n",
    "        # Parent name\n",
    "        parent_name = names[parent]\n",
    "        # Add tuple\n",
    "        tuples.append((parent_name, node_name))\n",
    "    return tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Module', 'Print'),\n",
       " ('Print', 'Str'),\n",
       " ('Print', 'bool'),\n",
       " ('Str', 'Hello World!')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_ast_pairs(get_ast_repr(ast.parse(sample_code)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# flatten = lambda l: [item for sublist in l for item in sublist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# flatten(get_ast_pairs(get_ast_repr(ast.parse(sample_code))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take the children only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Print', 'Str', 'bool', 'Hello World!']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " [pair[1] for pair in get_ast_pairs(get_ast_repr(ast.parse(sample_code)))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ast_docs = []\n",
    "ast_labels = []\n",
    "i = 0\n",
    "while i < len(docs):\n",
    "    doc = docs[i]\n",
    "    label = labels[i]\n",
    "    try:\n",
    "        ast_docs.append(\n",
    "            # flatten(get_ast_pairs(get_ast_repr(ast.parse(doc))))\n",
    "            [pair[1] for pair in get_ast_pairs(get_ast_repr(ast.parse(doc)))]\n",
    "        )\n",
    "        ast_labels.append(\n",
    "            label\n",
    "        )\n",
    "    except:\n",
    "        pass\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Number AST docs: 421,267'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Number AST docs: {:,}'.format(len(ast_docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ast_docs = [\n",
    "    ' '.join(array) for array in ast_docs\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Assign Assign Print Name Call Name Call BinOp bool a Store Name Call b Store Name Call Name Add Name int Load Name int Load Name a Load b Load raw_input Load raw_input Load'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ast_docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ast_t = Tokenizer(num_words=NUM_WORDS, \n",
    "                  lower=False, \n",
    "                  split=' ', \n",
    "                  char_level=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ast_t.fit_on_texts(ast_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: 10005368\n",
      "Load: 9607682\n",
      "Store: 2665169\n",
      "Call: 2205672\n",
      "Assign: 2186523\n"
     ]
    }
   ],
   "source": [
    "ordered_ast_nodes = sorted(ast_t.word_counts.iteritems(), key=lambda (k,v): (v,k), reverse=True)\n",
    "N = 5\n",
    "for key, value in ordered_ast_nodes[:N]:\n",
    "    print \"%s: %s\" % (key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i) Count: count of each tree leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoded_ast_count = ast_t.texts_to_matrix(ast_docs, mode='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  8.,  6., ...,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_ast_count[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/ast_count', encoded_ast_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.save('data/processed/ast_labels', ast_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}