16_Dependency_parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Natural language processing\n",
    "\n",
    "Practical course material for the ASDM Class 09 (Text Mining) by Florian Leitner.\n",
    "\n",
    "© 2016-2017 Florian Leitner. All rights reserved.\n",
    "\n",
    "## Installation and setup\n",
    "\n",
    "Most of the setup from yesterday for sequence tagging/entity extraction will be re-used today:  SpaCy, and NLTK's wrappers for Stanford CoreNLP libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.parse.corenlp import CoreNLPDependencyParser\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, we will directly use the CoreNLP API from Stanford, so you need to download the latest CoreNLP package and the two English language model JARs from [the CoreNLP website](https://stanfordnlp.github.io/CoreNLP/), put them all into a directory named `corenlp`, itself located in the same directory as this notebook, and then run the [CoreNLP Server](https://stanfordnlp.github.io/CoreNLP/corenlp-server.html).\n",
    "\n",
    "If you are using OSX or Linux and have `wget` installed, you can simply run the code below in a terminal to \"generate\" the required configuration described above:\n",
    "\n",
    "```bash\n",
    "wget http://nlp.stanford.edu/software/stanford-corenlp-full-2017-06-09.zip\n",
    "unzip stanford-corenlp-full-2017-06-09.zip\n",
    "cd stanford-corenlp-full-2017-06-09\n",
    "wget http://nlp.stanford.edu/software/stanford-english-corenlp-2017-06-09-models.jar\n",
    "wget http://nlp.stanford.edu/software/stanford-english-kbp-corenlp-2017-06-09-models.jar\n",
    "```\n",
    "\n",
    "As described in the CoreNLP documentation, to run the CoreNLP server, next execute (in that newly created directory):\n",
    "\n",
    "```bash\n",
    "java -mx4g -cp \"*\" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -timeout 15000 -port 9000\n",
    "```\n",
    "\n",
    "Timeout values are given in miliseconds, so we tell the parser to abort parsing after 15 seconds.\n",
    "The default port for the CoreNLP server is 9000 (and can therefore be left out from the above command).\n",
    "\n",
    "With the server up and running, [the NLTK corenlp API](http://www.nltk.org/api/nltk.parse.html?highlight=corenlpparser#module-nltk.parse.corenlp) simply acts as a client to the server API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corenlp = CoreNLPDependencyParser(\n",
    "    url='http://localhost:9000',\n",
    "    encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "example = 'The quick brown fox jumps over the lazy dog.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT4AAABlCAIAAABbZBjnAAAACXBIWXMAAA3XAAAN1wFCKJt4AAAAHXRFWHRTb2Z0d2FyZQBHUEwgR2hvc3RzY3JpcHQgOS4yMcb0+xQAABBdSURBVHic7Z2/j+PIlcdLh4u6DVhcQOPNulljJz2wA1I98EXTgKjgxqmo5AKvA1LxAjapP4Eab7gJecGuU3LTnYQcQBNcMk060wQ+kNN9l5xbsNgGrL5gb6ELnqe2TEls6rcovU/QEMlS8RWrvqxXr6pVlclkQhAEKRv/tGsDEARZBpQugpQSlC6ClBKULlKIIAgURdm1FcgPoHSRQgiCIMvyrq1AfqCCEWYEKSP/vGsDkL2m1+v5vs8O2+22ruuEkCAILMuSZbnX6xFCHMdxXRfOE0J0XU+SRBCEJElkWa7X67Ztp2nqeZ4sy6ZpRlHUbrfjOPY8j1KaJEmv11NVFe7ieZ5pmpTSNE1lWRYEAe6C/AMTBCmG7/uGYfBnGo0Gf8g3J0KIbduTyUTTtFarNZlMwjCED5PJxDCMarUKCSaTSRzHkiS5rguHoijGcQyfXdfN3AUBcKyLbAronwVB6HQ6hBBZltM0ZVdVVYUEhBBKqed5tm2zq0EQJEkCybDLnQlKF9kNgiDwh+Aew+cgCOI4Nk1TURRZlkHDSAYc6yK7IYoi/jAIAkopIQQEzHraJEkURVEUJSN1BHtdZHn4/nBRtzaKIl3XQahBEOi63u12CSGWZfGeMyiW97QRAKWLLI+qqoqimKYpy/JoNCKEmKZJPo5y2VAWQsqEkCiKIAFcrdfrkIPrukEQwLzxJ598Ap9N09R1XVGUbrcLHTLCg/O6SFE8zwvDMNO7RlHEpnCKZwUCzumo0zQFteMSrnngWBcpRJqmrutCrJhnQ0usBEFA0eaDDjOSh2malUqlUqnAyoq1yMk0zVevXr169apSqXiet3qGxwk6zAhSSrDXRZBSgtJFkFKCYSpk/SR3d8lwGN3ejv72t//6y1/+97vv/uWnP5XPzuTzc+H0dNfWHQg41kVWgldpdHubjsd/vL1lV8Va7f++//6/RyP+DK3V5LOzp0+eyOfn8vn5Dow+CFC6SFEeVSlokhDSfPZMOD1lsvSur/Wvv/7u++//7Ze/rJ6cRLe30c3N/cMDXJXOzmitRmu15rNntFajT55svWSlBKWLzGBplc4jHY/1r7/+Jgxb9brz2WfC6Wk6Hkc3N/5gkD48JMPhm/fvWeLGxYV8dvbJj36EPnYOKN1jJ7q5ScdjfzAghES3t8lw+GE4ZFeXUGkO0P0SQpzPPlMvL6ctSYbD8MMHeGswM9DHnglK94jIV6l0diacnq5LpfOY7n5zEgeDQTIcxnd36GNPg9I9TAqqFJzS7bf+/O53Huhj86B0S08wGBBC9lal81io+53HMfvYKN0yEQwG6cND+OEDdDu8D0n2WKU5LNf95nA8PjZKd0/JV2nj4kI4OaG12tMnT6BFlrctrqX7zcn8UH1slO7uWUilZWxkRVh79zuPg/GxUbrbA3oAcOeOWaXz2Gj3m0NJfWyU7kbgVZoMh+nDA++qkaNXaQ5b637nURYfG6W7KvkqrZ6cyOfntFYTTk7qoiicnCjPnu3Q2lKwq+53HvvpY6N0FwBVuk123v3msA8+Nkp3NplFvIQQVOn22bfudx478bFRuoWW2qNKd8g+d7/z2IKPfVzSXfs/xCDboSzdbw5r97EPVrqo0sOjjN3vPFb3sQ9TutQwpl0UgiotP6z7jXu9fZtoXZF5Pva8kh6mdJ1+n9ZqqNJDJbm7OzDdzgR8bP3qaubVw5Qughw8+GOuCFJKSindIAhghzhFUXDL80MFtvnLbMOLMMr3O8xJkpim6XkepTRJEtx59VBxHMc0TazfeZRSuoqiwIaruO0qcrSUyWEG0UKXq3yETxAEgaIosixTSlVV5fdcB+8L0kM+6GzvIeBMQSVO105O/RJCTNOE71JKIaWiKJk0B8WkbPi+bxjG9PkwDCVJiuOYJRNFcTQaweFoNJIkybZtONQ0TdO07RiMFMS2bU3TWJVpmiaKou/7cJhfv4ZhsO9CXZexbS9E+Yo3T7qtVisMQ/6MbduWZfFnNE2zLEvTNKZhZH/gpQhIksSkm1+/oijyl+I4PnjplslhzicIgswO65RS3/f5M71ez7KsJEl0Xd+udcjjUEoFQeDP8AOi/PrNRD2OIQhyONKVZTkzsEnTlK/sNE1hfEspRenuIdPj0iAI2Of8+s3MNRzDlNLhSLfT6ZimyQ7TNLVtu9PpsENVVTudjq7rjuMQQlC9+4aqqnwNOo7DazW/fnu9HgtKBUHApywvURQJgkApnTlDVqbJoSAILMtK0zRNU3it8nFIVVXhNawoCiTodrvgOEVRpKoqIcT3fV3XkyRJkuTNmzdRFB3D67ks9Ho90zShBpMkEQRB13XTNOGFm1O/hBBVVQVBgHlgSqnjOE+fPt1tcVYnTdP7+3v4kBlKkMNbw8xUnZk3QsoCvFgppTPHq8XrVxCEA1jOwTre6UuHJl0EIYSAw3zY8/aHM9ZFjhzwsSuVCviW3W531xZtFux1EaSUYK+LIKUEpYsgpaRMk0OP4l1f+4PBf/zpTz/79NOXv/iFenlZxp8ORHLwrq+/fPMmHY//9ec/71xdHcPP3MzjEMa6oFjv+vr+4eHTH//4f/7615/95Cf/+ec/E0Kks7P28+dqvX7MdXwAJHd3dr/vheGH4RCqGM5rL160Ly+P85exSyzd6ObGfffOefv2/uFBrNXUer39/Hl0c9P5wx8mX32V3N15Yei+ewe/4YoaLinBYOBeX//727fko1AJIc0vvhh9+aX17bcgZunsrHN1dWxOVvmkC4qFOquenKiXl81nz9iv8pqu64Vh8uoVS5+Ox9AtfxOGhBDp7Ey5uGg/f44/FrnPpOOx8/at3e9/GA7hvdz91a9AmcFg0Pzii8lXX0FKp993r6/fvH9fPTnRX7w4Hi+6NNLNVyxD+f3vCSHB7343nUNGw6yjRg3vFdHNjd3vw/CnVa+3nz/P1HJGukByd2e9fs2+1bm6Ongvet+lW1CxDGoYar3ea7dz8gQNhzc3UNOo4T3B6fftfv+Pt7f5/edM6QKZvrpzdaW/eHGoXvSeSjczUtVevMhXLKPym9/Yv/71vF+dnoYPcRV5NSBrB0JQELNoXFy0Ly/zqy9Hugzv+tp99+6bMIQ67b58eXhe9H5JN6PYVr0OQir44oRK9X/72yWcJdTw9uGHqerlZefqqojjU0S6wKIvhXKxF9JdUbEMp9+H8PIqxky76PXz82OLXm4UfqZHrNW6L18u9HiLSxeA8ZH1+vV0xKvU7FK6MDJZXbGM6fDyKvAaXot5SDAY2P0+hAmXnpJdVLr8F/l5poKd/N6yA+lOz9asa1IuJ7y8CqDh4P37db1ijo31Ro+Wli7Ae9HQ9krqRW9PutOKXfsaiSLh5VXAZR6LAjM90NHNnOlZghWlyyj7hPDGpbsFxTIWDS8vDWo4H6j0IjM9S7Au6QL8y6Vcyyo3KF0WoCdbadyrhJeXBpdqZdhCUHe90gXS8Zgtq1wicrYT1i9dfpYFAnrbcUXWEl5eGlyqlZnp2dxU6iakywBnoRRe9NqkO63YLTfc9YaXl+bYlmpNR3023V9tVLoAv6yycXEBhdrc7ZZjVenuXLGMDYWXV+Gwl3nwA6JtjhK3IF1gz5dVLindmf9wt9uOZdPh5VU4JA3vvEFvTbr8HWE6eqFVX5tmMeku+s8A22Rr4eVVmLlUa89tZvBLGnb43znbly6wb8sqF5Nu7/Vr69tv90qxADzWnff8xWEaJoTsfHxeEIgmbC3uOA+o6x26V/AfTsLp6W5HZ4tJNx2P98fXPwySu7u9jWFmwNrn2fnT2It/P0AQZFHwx1wRpJT8/cdc+Y1MGYIgyLIM++vxe+rtIZVKxff94luEQcqZpS6IaZpRFLXbbdzscwtsoRHCFpCEkF6vl9mDe0+Bze0JIY1Go9FoiKIoSVKj0YC/k4/wn/eQRqMRhuGiX1nxpoZhGIaxYiYIT36lbKERGobh+/6m77IWfvgJdeiCTNNsNpuKosB7bmdvlAVZpf9EkDLy97FuHMeZC4qiwO7vDNi2mO1NzF/yPI9SqigKpVTX9SX2Ne31eiwH0zQdx1EU5fPPP1cUhe1QDiczXrGu63By5ibXsCm2/BHTND3Pmy7pdLbFYY8FBhf8rZlh8Hxgy2a4GgSBoiiyLFNKVVVl27ezMrLiZM7AAwGnLqfgRZhpA7sdDATSNGXPh90op7rzS/2oMezrwHRD2mgjnAfbsHtmzso/wporKwtz8k3TXKWlZcn0wvMcBkKIZVnw2bbtVqvFLtm2rWnaaDSCQ9d1F3VsDMPgczAMo1qtMjMyuU3bPM/sOI5FUWSOdBzHkiQxFxeyDcOw0Wi4rruQwbyd7LFA/hkzYOgBpRuNRpA4DENJkuI4hjS+74uiyIrfaDTgcxzH8MG2bd4zJ4SwQ9d1+booTo4Nvu9n8pQkiZlXpLpnlroI+S1no40QmNmQ+DOWZWWKw240Go0yDcD3fU3TMonZM1+RotKVJGmmuZPJhG92LBPbtosbMZ0D/wiWlu60Ga7r8tn6vi9J0qKDZD7/TMVMN/pGozE9Hm61Wpmb2rbNGoRhGPAqYS+aTOkydVGtVpcwPt8GvoVlXhxFqntmqYuQr7eNNkL2rZntPwxD/yOZorHH2Gg0pu/IP8lpJa9C0e3CYLvhmaRpqqpq5uTTp0+L9/yU0kz+a3EqoihqNpv8mYyd4NtQSpe+xbTZzF9iZGwghARBkPHbKaWWZcF3m82m7/vgaoIHHgQBH1nN3PT+/n4Jy/NtaLfblmWBo+u6Lu/xFqzu6VKvzkYb4TxgzEUphbAzOPZ8Ajiv6/rM6YZut8ueJPuwFtaw0x+l1PO8nMf6KEmSpGnK5+B53up1TynNH/BEUQRDuyAIVrGfARurP5pMluUkSfhXRpqmbEJCURQIEDabzTAM11jZxW3Qdd2yrDRNoyiClwhLtnp1b4LNWaXruud57MkEQeD7/nQaQRBYdIA3A2oTXsGZJ7kia1iS0W63M11NEAQLhXx7vR6LgiRJoqoqX3gWwoGUCxkG7Y83LGOqruuO46iqulxUw3EcPkpkmman03n0W51OhzcjTVPbtvkvpmnqeZ6qqp1Ox7KstQU2FrEBuovMSbKO6s5BEAT+efJVn8/mrErTlOkNnlImAdyXtczpzp89yTVPSjPX2bIsfl6XRW4gkFOtVmEUNxqN4JAfaRiGAQMz+NBqtTIDj0eBUSLMLbuuyw85DMOAsRPcgnBBGtu2M9PRmQle27ZFUdQ0Db7OIhlQBFZAURRFUVxoeGYYBnyl0Wi0Wi045AMYcKlarTLD+GCYbdtQHE3TJEnKxMn4UbQoimwEFccxXxeTyaTVavEPZCHybRiNRqIozhx85lR3fqkfBepiOudNN8KchgRNiD0lVu/MYPJxTQQwM/Sw9OA/h3WuYYaXnCzLq/stbHoZDqMoAnduuZzXaNhMljMP3FGypoH9cqxiw4aeKjNpuZw3YRWYBIsLl8sBJg7X+6D29N8PMtJFkPLiOE4cx+tfwrneTnx1+CVcy023Isg+EMdxtVpljXmvHWYEQbYG/tMfgpQSlC6ClBKULoKUEpQugpQSlC6ClJL/B8p9EMv8PE4fAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy']), '.'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# parse_text returns a *list* of parsed sentences:\n",
    "corenlp_result, = corenlp.parse_text(example)\n",
    "corenlp_result.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SpaCy was set up yesterday, but to complete the \"proof of life\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "spacy_en = spacy.load(\"en\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "The quick brown fox jumps over the lazy dog."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_result = spacy_en(example)\n",
    "spacy_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing parse results\n",
    "\n",
    "Thanks to the NLTK wrapper, we are provided with a number of ways to visualize the CoreNLP parse result:\n",
    "\n",
    "As a simple (but very difficult!) tree/result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT4AAABlCAIAAABbZBjnAAAACXBIWXMAAA3XAAAN1wFCKJt4AAAAHXRFWHRTb2Z0d2FyZQBHUEwgR2hvc3RzY3JpcHQgOS4yMcb0+xQAABBdSURBVHic7Z2/j+PIlcdLh4u6DVhcQOPNulljJz2wA1I98EXTgKjgxqmo5AKvA1LxAjapP4Eab7gJecGuU3LTnYQcQBNcMk060wQ+kNN9l5xbsNgGrL5gb6ELnqe2TEls6rcovU/QEMlS8RWrvqxXr6pVlclkQhAEKRv/tGsDEARZBpQugpQSlC6ClBKULlKIIAgURdm1FcgPoHSRQgiCIMvyrq1AfqCCEWYEKSP/vGsDkL2m1+v5vs8O2+22ruuEkCAILMuSZbnX6xFCHMdxXRfOE0J0XU+SRBCEJElkWa7X67Ztp2nqeZ4sy6ZpRlHUbrfjOPY8j1KaJEmv11NVFe7ieZ5pmpTSNE1lWRYEAe6C/AMTBCmG7/uGYfBnGo0Gf8g3J0KIbduTyUTTtFarNZlMwjCED5PJxDCMarUKCSaTSRzHkiS5rguHoijGcQyfXdfN3AUBcKyLbAronwVB6HQ6hBBZltM0ZVdVVYUEhBBKqed5tm2zq0EQJEkCybDLnQlKF9kNgiDwh+Aew+cgCOI4Nk1TURRZlkHDSAYc6yK7IYoi/jAIAkopIQQEzHraJEkURVEUJSN1BHtdZHn4/nBRtzaKIl3XQahBEOi63u12CSGWZfGeMyiW97QRAKWLLI+qqoqimKYpy/JoNCKEmKZJPo5y2VAWQsqEkCiKIAFcrdfrkIPrukEQwLzxJ598Ap9N09R1XVGUbrcLHTLCg/O6SFE8zwvDMNO7RlHEpnCKZwUCzumo0zQFteMSrnngWBcpRJqmrutCrJhnQ0usBEFA0eaDDjOSh2malUqlUqnAyoq1yMk0zVevXr169apSqXiet3qGxwk6zAhSSrDXRZBSgtJFkFKCYSpk/SR3d8lwGN3ejv72t//6y1/+97vv/uWnP5XPzuTzc+H0dNfWHQg41kVWgldpdHubjsd/vL1lV8Va7f++//6/RyP+DK3V5LOzp0+eyOfn8vn5Dow+CFC6SFEeVSlokhDSfPZMOD1lsvSur/Wvv/7u++//7Ze/rJ6cRLe30c3N/cMDXJXOzmitRmu15rNntFajT55svWSlBKWLzGBplc4jHY/1r7/+Jgxb9brz2WfC6Wk6Hkc3N/5gkD48JMPhm/fvWeLGxYV8dvbJj36EPnYOKN1jJ7q5ScdjfzAghES3t8lw+GE4ZFeXUGkO0P0SQpzPPlMvL6ctSYbD8MMHeGswM9DHnglK94jIV6l0diacnq5LpfOY7n5zEgeDQTIcxnd36GNPg9I9TAqqFJzS7bf+/O53Huhj86B0S08wGBBC9lal81io+53HMfvYKN0yEQwG6cND+OEDdDu8D0n2WKU5LNf95nA8PjZKd0/JV2nj4kI4OaG12tMnT6BFlrctrqX7zcn8UH1slO7uWUilZWxkRVh79zuPg/GxUbrbA3oAcOeOWaXz2Gj3m0NJfWyU7kbgVZoMh+nDA++qkaNXaQ5b637nURYfG6W7KvkqrZ6cyOfntFYTTk7qoiicnCjPnu3Q2lKwq+53HvvpY6N0FwBVuk123v3msA8+Nkp3NplFvIQQVOn22bfudx478bFRuoWW2qNKd8g+d7/z2IKPfVzSXfs/xCDboSzdbw5r97EPVrqo0sOjjN3vPFb3sQ9TutQwpl0UgiotP6z7jXu9fZtoXZF5Pva8kh6mdJ1+n9ZqqNJDJbm7OzDdzgR8bP3qaubVw5Qughw8+GOuCFJKSindIAhghzhFUXDL80MFtvnLbMOLMMr3O8xJkpim6XkepTRJEtx59VBxHMc0TazfeZRSuoqiwIaruO0qcrSUyWEG0UKXq3yETxAEgaIosixTSlVV5fdcB+8L0kM+6GzvIeBMQSVO105O/RJCTNOE71JKIaWiKJk0B8WkbPi+bxjG9PkwDCVJiuOYJRNFcTQaweFoNJIkybZtONQ0TdO07RiMFMS2bU3TWJVpmiaKou/7cJhfv4ZhsO9CXZexbS9E+Yo3T7qtVisMQ/6MbduWZfFnNE2zLEvTNKZhZH/gpQhIksSkm1+/oijyl+I4PnjplslhzicIgswO65RS3/f5M71ez7KsJEl0Xd+udcjjUEoFQeDP8AOi/PrNRD2OIQhyONKVZTkzsEnTlK/sNE1hfEspRenuIdPj0iAI2Of8+s3MNRzDlNLhSLfT6ZimyQ7TNLVtu9PpsENVVTudjq7rjuMQQlC9+4aqqnwNOo7DazW/fnu9HgtKBUHApywvURQJgkApnTlDVqbJoSAILMtK0zRNU3it8nFIVVXhNawoCiTodrvgOEVRpKoqIcT3fV3XkyRJkuTNmzdRFB3D67ks9Ho90zShBpMkEQRB13XTNOGFm1O/hBBVVQVBgHlgSqnjOE+fPt1tcVYnTdP7+3v4kBlKkMNbw8xUnZk3QsoCvFgppTPHq8XrVxCEA1jOwTre6UuHJl0EIYSAw3zY8/aHM9ZFjhzwsSuVCviW3W531xZtFux1EaSUYK+LIKUEpYsgpaRMk0OP4l1f+4PBf/zpTz/79NOXv/iFenlZxp8ORHLwrq+/fPMmHY//9ec/71xdHcPP3MzjEMa6oFjv+vr+4eHTH//4f/7615/95Cf/+ec/E0Kks7P28+dqvX7MdXwAJHd3dr/vheGH4RCqGM5rL160Ly+P85exSyzd6ObGfffOefv2/uFBrNXUer39/Hl0c9P5wx8mX32V3N15Yei+ewe/4YoaLinBYOBeX//727fko1AJIc0vvhh9+aX17bcgZunsrHN1dWxOVvmkC4qFOquenKiXl81nz9iv8pqu64Vh8uoVS5+Ox9AtfxOGhBDp7Ey5uGg/f44/FrnPpOOx8/at3e9/GA7hvdz91a9AmcFg0Pzii8lXX0FKp993r6/fvH9fPTnRX7w4Hi+6NNLNVyxD+f3vCSHB7343nUNGw6yjRg3vFdHNjd3vw/CnVa+3nz/P1HJGukByd2e9fs2+1bm6Ongvet+lW1CxDGoYar3ea7dz8gQNhzc3UNOo4T3B6fftfv+Pt7f5/edM6QKZvrpzdaW/eHGoXvSeSjczUtVevMhXLKPym9/Yv/71vF+dnoYPcRV5NSBrB0JQELNoXFy0Ly/zqy9Hugzv+tp99+6bMIQ67b58eXhe9H5JN6PYVr0OQir44oRK9X/72yWcJdTw9uGHqerlZefqqojjU0S6wKIvhXKxF9JdUbEMp9+H8PIqxky76PXz82OLXm4UfqZHrNW6L18u9HiLSxeA8ZH1+vV0xKvU7FK6MDJZXbGM6fDyKvAaXot5SDAY2P0+hAmXnpJdVLr8F/l5poKd/N6yA+lOz9asa1IuJ7y8CqDh4P37db1ijo31Ro+Wli7Ae9HQ9krqRW9PutOKXfsaiSLh5VXAZR6LAjM90NHNnOlZghWlyyj7hPDGpbsFxTIWDS8vDWo4H6j0IjM9S7Au6QL8y6Vcyyo3KF0WoCdbadyrhJeXBpdqZdhCUHe90gXS8Zgtq1wicrYT1i9dfpYFAnrbcUXWEl5eGlyqlZnp2dxU6iakywBnoRRe9NqkO63YLTfc9YaXl+bYlmpNR3023V9tVLoAv6yycXEBhdrc7ZZjVenuXLGMDYWXV+Gwl3nwA6JtjhK3IF1gz5dVLindmf9wt9uOZdPh5VU4JA3vvEFvTbr8HWE6eqFVX5tmMeku+s8A22Rr4eVVmLlUa89tZvBLGnb43znbly6wb8sqF5Nu7/Vr69tv90qxADzWnff8xWEaJoTsfHxeEIgmbC3uOA+o6x26V/AfTsLp6W5HZ4tJNx2P98fXPwySu7u9jWFmwNrn2fnT2It/P0AQZFHwx1wRpJT8/cdc+Y1MGYIgyLIM++vxe+rtIZVKxff94luEQcqZpS6IaZpRFLXbbdzscwtsoRHCFpCEkF6vl9mDe0+Bze0JIY1Go9FoiKIoSVKj0YC/k4/wn/eQRqMRhuGiX1nxpoZhGIaxYiYIT36lbKERGobh+/6m77IWfvgJdeiCTNNsNpuKosB7bmdvlAVZpf9EkDLy97FuHMeZC4qiwO7vDNi2mO1NzF/yPI9SqigKpVTX9SX2Ne31eiwH0zQdx1EU5fPPP1cUhe1QDiczXrGu63By5ibXsCm2/BHTND3Pmy7pdLbFYY8FBhf8rZlh8Hxgy2a4GgSBoiiyLFNKVVVl27ezMrLiZM7AAwGnLqfgRZhpA7sdDATSNGXPh90op7rzS/2oMezrwHRD2mgjnAfbsHtmzso/wporKwtz8k3TXKWlZcn0wvMcBkKIZVnw2bbtVqvFLtm2rWnaaDSCQ9d1F3VsDMPgczAMo1qtMjMyuU3bPM/sOI5FUWSOdBzHkiQxFxeyDcOw0Wi4rruQwbyd7LFA/hkzYOgBpRuNRpA4DENJkuI4hjS+74uiyIrfaDTgcxzH8MG2bd4zJ4SwQ9d1+booTo4Nvu9n8pQkiZlXpLpnlroI+S1no40QmNmQ+DOWZWWKw240Go0yDcD3fU3TMonZM1+RotKVJGmmuZPJhG92LBPbtosbMZ0D/wiWlu60Ga7r8tn6vi9J0qKDZD7/TMVMN/pGozE9Hm61Wpmb2rbNGoRhGPAqYS+aTOkydVGtVpcwPt8GvoVlXhxFqntmqYuQr7eNNkL2rZntPwxD/yOZorHH2Gg0pu/IP8lpJa9C0e3CYLvhmaRpqqpq5uTTp0+L9/yU0kz+a3EqoihqNpv8mYyd4NtQSpe+xbTZzF9iZGwghARBkPHbKaWWZcF3m82m7/vgaoIHHgQBH1nN3PT+/n4Jy/NtaLfblmWBo+u6Lu/xFqzu6VKvzkYb4TxgzEUphbAzOPZ8Ajiv6/rM6YZut8ueJPuwFtaw0x+l1PO8nMf6KEmSpGnK5+B53up1TynNH/BEUQRDuyAIVrGfARurP5pMluUkSfhXRpqmbEJCURQIEDabzTAM11jZxW3Qdd2yrDRNoyiClwhLtnp1b4LNWaXruud57MkEQeD7/nQaQRBYdIA3A2oTXsGZJ7kia1iS0W63M11NEAQLhXx7vR6LgiRJoqoqX3gWwoGUCxkG7Y83LGOqruuO46iqulxUw3EcPkpkmman03n0W51OhzcjTVPbtvkvpmnqeZ6qqp1Ox7KstQU2FrEBuovMSbKO6s5BEAT+efJVn8/mrErTlOkNnlImAdyXtczpzp89yTVPSjPX2bIsfl6XRW4gkFOtVmEUNxqN4JAfaRiGAQMz+NBqtTIDj0eBUSLMLbuuyw85DMOAsRPcgnBBGtu2M9PRmQle27ZFUdQ0Db7OIhlQBFZAURRFUVxoeGYYBnyl0Wi0Wi045AMYcKlarTLD+GCYbdtQHE3TJEnKxMn4UbQoimwEFccxXxeTyaTVavEPZCHybRiNRqIozhx85lR3fqkfBepiOudNN8KchgRNiD0lVu/MYPJxTQQwM/Sw9OA/h3WuYYaXnCzLq/stbHoZDqMoAnduuZzXaNhMljMP3FGypoH9cqxiw4aeKjNpuZw3YRWYBIsLl8sBJg7X+6D29N8PMtJFkPLiOE4cx+tfwrneTnx1+CVcy023Isg+EMdxtVpljXmvHWYEQbYG/tMfgpQSlC6ClBKULoKUEpQugpQSlC6ClJL/B8p9EMv8PE4fAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy']), '.'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corenlp_result.tree()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fully dependency-label annotated tree; Notice the tricky attachment of the word \"over\" (discussed below, in the Comparison section)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"467pt\" height=\"302pt\"\n",
       " viewBox=\"0.00 0.00 467.37 302.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 298)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-298 463.3657,-298 463.3657,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"316.7949\" y=\"-271.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0 (None)</text>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"316.7949\" y=\"-185.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">5 (jumps)</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;5 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M316.7949,-257.7616C316.7949,-246.3597 316.7949,-231.4342 316.7949,-218.494\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"320.295,-218.2121 316.7949,-208.2121 313.295,-218.2121 320.295,-218.2121\"/>\n",
       "<text text-anchor=\"middle\" x=\"335.8501\" y=\"-228.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">ROOT</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"152.7949\" y=\"-99.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">4 (fox)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;4 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>5&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M282.4151,-171.9716C255.0739,-157.6341 216.9235,-137.6284 189.0926,-123.0341\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"190.5621,-119.8527 180.0804,-118.3083 187.3112,-126.0521 190.5621,-119.8527\"/>\n",
       "<text text-anchor=\"middle\" x=\"259.9639\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">nsubj</text>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"316.7949\" y=\"-99.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">9 (dog)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;9 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M316.7949,-171.7616C316.7949,-160.3597 316.7949,-145.4342 316.7949,-132.494\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"320.295,-132.2121 316.7949,-122.2121 313.295,-132.2121 320.295,-132.2121\"/>\n",
       "<text text-anchor=\"middle\" x=\"332.7397\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">nmod</text>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"389.7949\" y=\"-99.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">10 (.)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;10 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M336.0249,-171.8216C341.6111,-166.2649 347.6022,-160.0263 352.7949,-154 359.2141,-146.5504 365.7481,-138.0741 371.5031,-130.2441\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"374.3816,-132.2363 377.3999,-122.0805 368.7071,-128.1375 374.3816,-132.2363\"/>\n",
       "<text text-anchor=\"middle\" x=\"379.3467\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">punct</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"28.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">1 (The)</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"108.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">2 (quick)</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"195.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">3 (brown)</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>4&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M126.8005,-85.9716C108.2827,-73.1286 83.2073,-55.7376 63.0563,-41.7619\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"64.967,-38.8277 54.7552,-36.0047 60.9777,-44.5797 64.967,-38.8277\"/>\n",
       "<text text-anchor=\"middle\" x=\"107.3467\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">det</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;2 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M143.4637,-85.7616C137.4551,-74.0176 129.534,-58.5355 122.7802,-45.3349\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"125.7834,-43.5204 118.1127,-36.2121 119.5517,-46.7088 125.7834,-43.5204\"/>\n",
       "<text text-anchor=\"middle\" x=\"149.3467\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">amod</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;3 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>4&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M161.9141,-85.7616C167.7861,-74.0176 175.5272,-58.5355 182.1275,-45.3349\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"185.3472,-46.7216 186.6889,-36.2121 179.0862,-43.5911 185.3472,-46.7216\"/>\n",
       "<text text-anchor=\"middle\" x=\"193.3467\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">amod</text>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"279.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">6 (over)</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;6 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>9&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M308.9482,-85.7616C303.9446,-74.1316 297.3638,-58.8357 291.721,-45.72\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"294.7976,-44.0148 287.6304,-36.2121 288.3674,-46.7813 294.7976,-44.0148\"/>\n",
       "<text text-anchor=\"middle\" x=\"312.8398\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">case</text>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"354.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">7 (the)</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;7 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>9&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M324.8537,-85.7616C329.9926,-74.1316 336.7512,-58.8357 342.5466,-45.72\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"345.9074,-46.7736 346.7477,-36.2121 339.5046,-43.9444 345.9074,-46.7736\"/>\n",
       "<text text-anchor=\"middle\" x=\"347.3467\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">det</text>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"429.7949\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">8 (lazy)</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;8 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M340.4834,-85.9716C357.2078,-73.2433 379.8019,-56.0478 398.0799,-42.1371\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"400.2997,-44.8461 406.1376,-36.0047 396.0604,-39.2758 400.2997,-44.8461\"/>\n",
       "<text text-anchor=\"middle\" x=\"396.3467\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">amod</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<DependencyGraph with 11 nodes>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corenlp_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very compact and rather standard format, \"OTPL\" (one token per line), is the 4 column [CONLL Univ. format](http://universaldependencies.org/format.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The\tDT\t4\tdet\n",
      "quick\tJJ\t4\tamod\n",
      "brown\tJJ\t4\tamod\n",
      "fox\tNN\t5\tnsubj\n",
      "jumps\tVBZ\t0\tROOT\n",
      "over\tIN\t9\tcase\n",
      "the\tDT\t9\tdet\n",
      "lazy\tJJ\t9\tamod\n",
      "dog\tNN\t5\tnmod\n",
      ".\t.\t5\tpunct\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(corenlp_result.to_conll(4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use that to generate our own output that we can easily compare with SpaCy's result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def showcore(sentence):\n",
    "    print(\"CoreNLP   Token  PoS -dep-> Head\\n\")\n",
    "    row = \"{: >15s} {: >4s} -{}-> {}\"\n",
    "    lines = sentence.to_conll(4).strip().split('\\n')\n",
    "    words = [line.split('\\t') for line in lines]\n",
    "    tokens = [w[0] for w in words]\n",
    "    \n",
    "    for idx, word in enumerate(words):\n",
    "        token, pos, dep, tag = word\n",
    "        print(row.format(\n",
    "            token, pos, tag, tokens[int(dep)-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoreNLP   Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> fox\n",
      "          quick   JJ -amod-> fox\n",
      "          brown   JJ -amod-> fox\n",
      "            fox   NN -nsubj-> jumps\n",
      "          jumps  VBZ -ROOT-> .\n",
      "           over   IN -case-> dog\n",
      "            the   DT -det-> dog\n",
      "           lazy   JJ -amod-> dog\n",
      "            dog   NN -nmod-> jumps\n",
      "              .    . -punct-> jumps\n"
     ]
    }
   ],
   "source": [
    "showcore(corenlp_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need a simple way to visualize the SpaCy results; SpaCy also has a visualizer called [displaCy](https://explosion.ai/blog/displacy-js-nlp-visualizer), but it requires JavaScript and Node.js installed to set up, which is well beyond the scope of this class (hopefully a pure Python solution for Notebooks will come one day...). And the demo doesn't seem to run consistently..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def showcy(sentence):\n",
    "    print(\"SpaCy     Token  PoS -dep-> Head\\n\")\n",
    "    row = \"{: >15s} {: >4s} -{}-> {}\"\n",
    "    \n",
    "    for token in sentence:\n",
    "        print(row.format(\n",
    "            str(token),\n",
    "            token.tag_,\n",
    "            token.dep_,\n",
    "            # make this look like CoreNLP's output:\n",
    "            token.head if token.head != token else \".\"))\n",
    "    \n",
    "    if sentence.noun_chunks:\n",
    "        print(\"\\nNOUN PHRASES:\")\n",
    "        for chunk in sentence.noun_chunks:\n",
    "            print(chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SpaCy     Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> jumps\n",
      "          quick   JJ -amod-> jumps\n",
      "          brown   JJ -amod-> jumps\n",
      "            fox   NN -nsubj-> jumps\n",
      "          jumps  VBZ -ROOT-> .\n",
      "           over   IN -prep-> jumps\n",
      "            the   DT -det-> dog\n",
      "           lazy   JJ -amod-> dog\n",
      "            dog   NN -pobj-> over\n",
      "              .    . -punct-> jumps\n",
      "\n",
      "NOUN PHRASES:\n",
      "fox\n",
      "the lazy dog\n"
     ]
    }
   ],
   "source": [
    "showcy(spacy_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please refer to the documentaiton for \"decyphering\" [(Penn Treebank) Part-of-Speech](http://www.clips.ua.ac.be/pages/mbsp-tags) and [(Universal) dependency labels](http://universaldependencies.org/u/dep/index.html); Note that SpaCy still is using the older [Stanford dependency labels](https://nlp.stanford.edu/software/dependencies_manual.pdf) (in case you are not finding a label among the Universal dependency labels).\n",
    "\n",
    "## Comparing SpaCy and CoreNLP\n",
    "\n",
    "Finally, lets run some comparisons of the parse results.\n",
    "\n",
    "We will just look at some qualitative results.\n",
    "Note that this shouldn't be taken as the final word on this matter and is just here for educative purposes to teach students how to \"read\" dependency parse results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoreNLP   Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> fox\n",
      "          quick   JJ -amod-> fox\n",
      "          brown   JJ -amod-> fox\n",
      "            fox   NN -nsubj-> jumps\n",
      "          jumps  VBZ -ROOT-> .\n",
      "           over   IN -case-> dog\n",
      "            the   DT -det-> dog\n",
      "           lazy   JJ -amod-> dog\n",
      "            dog   NN -nmod-> jumps\n",
      "              .    . -punct-> jumps\n"
     ]
    }
   ],
   "source": [
    "showcore(corenlp_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SpaCy     Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> jumps\n",
      "          quick   JJ -amod-> jumps\n",
      "          brown   JJ -amod-> jumps\n",
      "            fox   NN -nsubj-> jumps\n",
      "          jumps  VBZ -ROOT-> .\n",
      "           over   IN -prep-> jumps\n",
      "            the   DT -det-> dog\n",
      "           lazy   JJ -amod-> dog\n",
      "            dog   NN -pobj-> over\n",
      "              .    . -punct-> jumps\n",
      "\n",
      "NOUN PHRASES:\n",
      "fox\n",
      "the lazy dog\n"
     ]
    }
   ],
   "source": [
    "showcy(spacy_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SpaCy got the attachment of \"over\" wrong.\n",
    "This is probsably due to it using the old Stanford depencencies, while the new Universal dependencies provide a new attachment mechanism for prepositions, using a new [`case` dependency](http://universaldependencies.org/u/dep/case.html).\n",
    "In addition, it did miss the correct attachment of the tokens \"The quick brown\": Each should be attached to the noun \"fox\".\n",
    "Admittedly, SpaCy messed this \"classical\" example sentence up pretty badly, if compred to CoreNLP.\n",
    "\n",
    "Going forward, let's simplify this comparison effort:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compare(sentence):\n",
    "    corenlp_parse, = corenlp.parse_text(sentence)\n",
    "    spacy_parse = spacy_en(sentence)\n",
    "    showcore(corenlp_parse)\n",
    "    print('---------------------------------')\n",
    "    showcy(spacy_parse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoreNLP   Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> Nexus\n",
      "         Google  NNP -compound-> Nexus\n",
      "          Nexus  NNP -nsubj-> challenges\n",
      "     challenges  VBZ -ROOT-> .\n",
      "            the   DT -det-> iPhone\n",
      "         iPhone  NNP -nmod:poss-> dominance\n",
      "             's  POS -case-> iPhone\n",
      "      dominance   NN -dobj-> challenges\n",
      "              .    . -punct-> challenges\n",
      "---------------------------------\n",
      "SpaCy     Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> challenges\n",
      "         Google  NNP -compound-> Nexus\n",
      "          Nexus  NNP -nsubj-> challenges\n",
      "     challenges  VBZ -ROOT-> .\n",
      "            the   DT -det-> iPhone\n",
      "         iPhone  NNP -poss-> dominance\n",
      "             's  POS -case-> iPhone\n",
      "      dominance   NN -dobj-> challenges\n",
      "              .    . -punct-> challenges\n",
      "\n",
      "NOUN PHRASES:\n",
      "Google Nexus\n",
      "the iPhone's dominance\n"
     ]
    }
   ],
   "source": [
    "compare(\"The Google Nexus challenges the iPhone's dominance.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CoreNLP parse is perfect here, no need for comments. SpaCy seems to have the problems again with an initial determiner, similar to the case in first sentence, but otherwise is quite good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoreNLP   Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> Sachs\n",
      "      economist   NN -compound-> Sachs\n",
      "        Jeffrey  NNP -compound-> Sachs\n",
      "          Sachs  NNP -nsubj-> coined\n",
      "         coined  VBD -ROOT-> .\n",
      "            the   DT -det-> therapy\n",
      "           term   NN -compound-> therapy\n",
      "             ``   `` -punct-> therapy\n",
      "          shock   NN -compound-> therapy\n",
      "        therapy   NN -dobj-> coined\n",
      "             ''   '' -punct-> therapy\n",
      "              .    . -punct-> coined\n",
      "---------------------------------\n",
      "SpaCy     Token  PoS -dep-> Head\n",
      "\n",
      "            The   DT -det-> economist\n",
      "      economist   NN -nsubj-> coined\n",
      "        Jeffrey  NNP -compound-> Sachs\n",
      "          Sachs  NNP -nsubj-> coined\n",
      "         coined  VBD -ROOT-> .\n",
      "            the   DT -det-> term\n",
      "           term   NN -dobj-> coined\n",
      "              \"   `` -punct-> term\n",
      "          shock   NN -compound-> therapy\n",
      "        therapy   NN -appos-> term\n",
      "              \"   '' -punct-> term\n",
      "              .    . -punct-> coined\n",
      "\n",
      "NOUN PHRASES:\n",
      "The economist\n",
      "Jeffrey Sachs\n",
      "the term\n"
     ]
    }
   ],
   "source": [
    "compare('The economist Jeffrey Sachs coined the term \"shock therapy\".')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, SpaCy clearly made the better attachement of the long noun phrase \"The economist Jeffrey Sachs\", and while CoreNLP treats 'the term \"shock therapy\"' (which is slightly unusual due to containing punctuation tokens) as a single phrase, SpaCy split it in two, which is technically more correct: We can say both \"...Sachs coined the term.\" and '...Sachs coined \"shock therapy\".' without depending on the other phrase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CoreNLP   Token  PoS -dep-> Head\n",
      "\n",
      "          India  NNP -nsubj-> purchasing\n",
      "           will   MD -aux-> purchasing\n",
      "             be   VB -aux-> purchasing\n",
      "     purchasing  VBG -ROOT-> .\n",
      "       Predator  NNP -compound-> drones\n",
      "         drones  NNS -dobj-> purchasing\n",
      "              ,    , -punct-> drones\n",
      "              a   DT -det-> source\n",
      "         source   NN -appos-> drones\n",
      "             at   IN -case-> Martin\n",
      "       Lockheed  NNP -compound-> Martin\n",
      "         Martin  NNP -nmod-> source\n",
      "       confirms  VBZ -advcl-> purchasing\n",
      "              .    . -punct-> purchasing\n",
      "---------------------------------\n",
      "SpaCy     Token  PoS -dep-> Head\n",
      "\n",
      "          India  NNP -nsubj-> purchasing\n",
      "           will   MD -aux-> purchasing\n",
      "             be   VB -aux-> purchasing\n",
      "     purchasing  VBG -ROOT-> .\n",
      "       Predator  NNP -compound-> drones\n",
      "         drones  NNS -dobj-> purchasing\n",
      "              ,    , -punct-> drones\n",
      "              a   DT -det-> source\n",
      "         source   NN -nsubj-> confirms\n",
      "             at   IN -prep-> source\n",
      "       Lockheed  NNP -compound-> Martin\n",
      "         Martin  NNP -pobj-> at\n",
      "       confirms  VBZ -advcl-> purchasing\n",
      "              .    . -punct-> purchasing\n",
      "\n",
      "NOUN PHRASES:\n",
      "India\n",
      "Predator drones\n",
      "a source\n",
      "Lockheed Martin\n"
     ]
    }
   ],
   "source": [
    "compare(\"India will be purchasing Predator drones, a source at Lockheed Martin confirms.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, SpaCy generated the better parse, correctly attaching \"source\" to \"confirms\" (and not \"drones\").\n",
    "Note the beauty of this last example: The parse clearly shows us how \"a source at Lockheed Martin confirms\" is an (adverbial) clause that modifies the purchase (action).\n",
    "Therefore, from the SpaCy parse, it is easy to extract \"India will be purching Predator drones\" as the main clause, and that the second clause can probably be dropped when trying to detect the \"essence\" of the sentence (e.g., for summarization).\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "Overall, SpaCy's parsing facilities are quite impressive for long, technical sentences as they are typical in articles and non-fiction.\n",
    "Stanford clearly has the advantage over for short, natural phrases, as found in speech and ficition/novels.\n",
    "However, Stanford has released several experimental parsers that will probably soon catch up again with SpaCy's performance on the long phrases (and hopefully without hurting their excellent performance on short expressions...)\n",
    "\n",
    "Overall, notice that perfect parsing still is a rather difficult task, and that the performance in other languages degrades rapidly, particulary if the sentence structure gets more complex (like in German...; not shown here).\n",
    "\n",
    "If you only need to parse English languauge, another high-performing and *fast* (!) parser is [NLP4J](https://emorynlp.github.io/nlp4j/) (formerly ClearNLP) parser, also written in Java. Finally, there is [SyntaxNet](https://github.com/tensorflow/models/tree/master/syntaxnet), Google's recently published neural parser built on TensorFlow, that claims to beat them all (but is relatively slow on in inference mode...).\n",
    "One option is to run all parsers and pick the best attachment (e.g., via [boosting](https://en.wikipedia.org/wiki/Boosting_%28machine_learning%29), [stacking](https://en.wikipedia.org/wiki/Ensemble_learning#Stacking), or other [ensemble methods](http://scikit-learn.org/stable/modules/ensemble.html)), and you might get results that are [even] close[r] to perfect..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}