zQuestionable - Part 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering with PyTorch Transformers: Part 1\n",
    "\n",
    "Article for this notebook: https://medium.com/@patonw/question-answering-with-pytorch-transformers-part-1-8736196bf20e\n",
    "\n",
    "> In the first part of this series we’ll look at the problem of question answering and the SQUAD datasets. Then we’ll see how the Transformers’ pipeline API allows us to easily use pre-trained models to answer questions.\n",
    ">\n",
    "> In later parts we’ll explore how to build systems around it that are generally useful to the average person. There have been many academic articles written on the topics we’ll explore. However, I want to focus on the engineering aspects and demonstrate how simple it is to build useful systems by leveraging a handful of high quality open-source libraries.\n",
    "\n",
    "Skip ahead to Part 2 & 3 for the meat.\n",
    "\n",
    "This is just an exploration of the SQUAD2.0 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare to run in paperspace. You should manage these with pipenv or conda on your own machine.\n",
    "# Run init_container from a Terminal window for debugging\n",
    "# I'd rather not have the output filling up the screen here.\n",
    "%run init_container.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qa.constants import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(SQUAD_TRAIN) as f:\n",
    "    doc = json.load(f)\n",
    "doc.keys(), type(doc[\"data\"]), len(doc[\"data\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc[\"data\"][0].keys(), doc[\"data\"][0][\"title\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(doc[\"data\"][0][\"paragraphs\"]), doc[\"data\"][0][\"paragraphs\"][0].keys(), len(doc[\"data\"][0][\"paragraphs\"][0][\"qas\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc[\"data\"][0][\"paragraphs\"][0][\"context\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc[\"data\"][0][\"paragraphs\"][0][\"qas\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraphs = []\n",
    "questions = []\n",
    "impossible = []\n",
    "for topic in doc[\"data\"]:\n",
    "    for pgraph in topic[\"paragraphs\"]:\n",
    "        paragraphs.append(pgraph[\"context\"])\n",
    "        for qa in pgraph[\"qas\"]:\n",
    "            if not qa[\"is_impossible\"]:\n",
    "                questions.append((qa[\"question\"], pgraph[\"context\"]))\n",
    "            else:\n",
    "                impossible.append((qa[\"question\"], pgraph[\"context\"]))\n",
    "        \n",
    "len(paragraphs), len(questions), len(impossible)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.sample(paragraphs, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.sample(questions, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.sample(impossible, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}