Day4_spacy_modular_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The purpose of this notebook is to demonstrate the value of modular pipelines and interact with them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lots of functionality in spacy is modular : Loadable and configurable modules.  This includes languages, models, alternative processors, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why do we care about modular text processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Legos | Brick Wall\n",
    "- | - \n",
    "![alt](images/legos.jpg) | ![alt](images/brickwall.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# There are many languages besides English supported by spacy:\n",
    "## Including: German, French, Spanish, \n",
    "## https://spacy.io/usage/models#languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# There's also an entire ecosystem of additional models, functionality that is modular within spacy.  Let's look at a few here:\n",
    "## https://spacy.io/universe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's start playing with NLP legos... First we will set up a default pipeline -- notice there are no arguments when calling load()...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('tagger', <spacy.pipeline.pipes.Tagger object at 0x7f374e66a390>)\n",
      "('parser', <spacy.pipeline.pipes.DependencyParser object at 0x7f37170cc528>)\n",
      "('ner', <spacy.pipeline.pipes.EntityRecognizer object at 0x7f37170cc588>)\n"
     ]
    }
   ],
   "source": [
    "for pipe in default_nlp.pipeline:\n",
    "    print(pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now set up a pipeline where some steps are not enabled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "simpler_nlp = spacy.load(\"en_core_web_sm\", disable=[\"parser\", \"tagger\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('ner', <spacy.pipeline.pipes.EntityRecognizer object at 0x7f37126a21c8>)\n"
     ]
    }
   ],
   "source": [
    "for pipe in simpler_nlp.pipeline:\n",
    "    print(pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Which will be faster?  Let's experiment..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_text = \"There is slight enlargement of the spleen. No history of a heart murmur.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of our document set : 10\n"
     ]
    }
   ],
   "source": [
    "MAX_DOCUMENTS = 10\n",
    "\n",
    "document_set = []\n",
    "for i in range(MAX_DOCUMENTS):\n",
    "    # add this document in N times...\n",
    "    document_set.append(example_text)\n",
    "    \n",
    "print('Size of our document set : {}'.format(len(document_set)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 181 ms, sys: 1.19 ms, total: 183 ms\n",
      "Wall time: 181 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for text in document_set:\n",
    "    default_nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 88.8 ms, sys: 598 µs, total: 89.4 ms\n",
      "Wall time: 87 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for text in document_set:\n",
    "    simpler_nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's pause for a moment and try what we did above but instead of running the pipeline over 10 documents, let's run it over 1000 documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'CHANGE_ME' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-b3aadf68dae0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mMAX_DOCUMENTS\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCHANGE_ME\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mlarger_document_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mMAX_DOCUMENTS\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0;31m# add this document in N times...\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'CHANGE_ME' is not defined"
     ]
    }
   ],
   "source": [
    "MAX_DOCUMENTS = CHANGE_ME\n",
    "\n",
    "larger_document_set = []\n",
    "for i in range(MAX_DOCUMENTS):\n",
    "    # add this document in N times...\n",
    "    larger_document_set.append(example_text)\n",
    "    \n",
    "print('Size of our larger document set : {}'.format(len(larger_document_set)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "for text in larger_document_set:\n",
    "    default_nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "for text in larger_document_set:\n",
    "    default_nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Since the steps of a pipeline are modular, let's change the order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_text = u\"This is a sentence.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_component(doc):\n",
    "    print(\"After tokenization, this doc has {} tokens.\".format(len(doc)))\n",
    "    print(\"The part-of-speech tags are:\", [token.pos_ for token in doc])\n",
    "    if len(doc) < 10:\n",
    "        print(\"This is a pretty short document.\")\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_pipeline = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "custom_pipeline.add_pipe(my_component, name=\"print_info\", last=True)\n",
    "\n",
    "print(custom_pipeline.pipe_names)\n",
    "\n",
    "doc = custom_pipeline(simple_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What happens when we try to print part of speech tags as the first step in the pipeline?  Change the code below to run the \"print_info\" component as the first component instead of the last"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_first_pipeline = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "print_first_pipeline.add_pipe(my_component, name=\"print_info\", first = CHANGE_ME)\n",
    "\n",
    "print(print_first_pipeline.pipe_names)\n",
    "\n",
    "doc = print_first_pipeline(simple_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's look at the components in this pipeline again.  Change the code below so that for each of the pipelines above (custom_pipeline and print_first_pipeline) we write out their steps and the order they are executed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pipe in CHANGE_ME.pipeline:\n",
    "    print(pipe)\n",
    "    \n",
    "for pipe in CHANGE_ME_TOO.pipeline:\n",
    "    print(pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How many steps do they each have?  What is the difference between them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}