docs/passim_quickstart.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyN2jvMe3OgvDuXO7q1h4Di1",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dasmiq/passim/blob/main/docs/passim_quickstart.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Getting started with `passim`\n",
        "\n",
        "The [Passim package](https://github.com/dasmiq/passim) is a python library and set of command-line tools for analyzing document for test reuse. This notebook walks through the simplest method of installing it and presents a couple of case studies with a small corpus. One of the main reasons to use Passim, however, is its scalability to larger corpora by taking advantage of clusters of machines using the [Apache Spark](https://spark.apache.org/) library. Although we don't discuss issues with these large deployments here, we do show a simple example of using Spark to manipulate Passim's output.\n",
        "\n",
        "Since Spark uses java code at runtime, we check that we have a working `java` executable."
      ],
      "metadata": {
        "id": "Wp36FnXm4wMv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!java --version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Rqsmo95I4LfN",
        "outputId": "bce9df60-8455-4d9b-b6f8-a127205a6af0"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "openjdk 11.0.24 2024-07-16\n",
            "OpenJDK Runtime Environment (build 11.0.24+8-post-Ubuntu-1ubuntu322.04)\n",
            "OpenJDK 64-Bit Server VM (build 11.0.24+8-post-Ubuntu-1ubuntu322.04, mixed mode, sharing)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The easiest way to install Passim is to have python's `pip` tool download it directly from the GitHub repository."
      ],
      "metadata": {
        "id": "ZsOJOl7p6Muq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install git+https://github.com/dasmiq/passim.git\n",
        "# If you're on a shared machine, use --user to install in your home directory.\n",
        "# !pip install --user git+https://github.com/dasmiq/passim.git"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nltlHSscJyrD",
        "outputId": "361179bf-4e2f-415c-8b8d-01b4c2fcbfe7"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting git+https://github.com/dasmiq/passim.git\n",
            "  Cloning https://github.com/dasmiq/passim.git to /tmp/pip-req-build-ufgvgeq9\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/dasmiq/passim.git /tmp/pip-req-build-ufgvgeq9\n",
            "  Resolved https://github.com/dasmiq/passim.git to commit 398144f146654372a211e256deae242602a4b3d8\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pyspark>=3.0.1 (from passim==2.0.0)\n",
            "  Downloading pyspark-3.5.2.tar.gz (317.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.3/317.3 MB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting graphframes (from passim==2.0.0)\n",
            "  Downloading graphframes-0.6-py2.py3-none-any.whl.metadata (934 bytes)\n",
            "Collecting intervaltree (from passim==2.0.0)\n",
            "  Downloading intervaltree-3.1.0.tar.gz (32 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: py4j==0.10.9.7 in /usr/local/lib/python3.10/dist-packages (from pyspark>=3.0.1->passim==2.0.0) (0.10.9.7)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from graphframes->passim==2.0.0) (1.26.4)\n",
            "Collecting nose (from graphframes->passim==2.0.0)\n",
            "  Downloading nose-1.3.7-py3-none-any.whl.metadata (1.7 kB)\n",
            "Requirement already satisfied: sortedcontainers<3.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from intervaltree->passim==2.0.0) (2.4.0)\n",
            "Downloading graphframes-0.6-py2.py3-none-any.whl (18 kB)\n",
            "Downloading nose-1.3.7-py3-none-any.whl (154 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m154.7/154.7 kB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: passim, pyspark, intervaltree\n",
            "  Building wheel for passim (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for passim: filename=passim-2.0.0-py3-none-any.whl size=14808 sha256=31dc0fed66188f60f24ed5c07c0a1f3d7c2b0290a24bb328521ea7c3a6d90b47\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-s4yms_ti/wheels/12/f4/5b/994acfd9665a3e2a485636f9dd808d7c51cf2d6c1a2e0475b5\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.5.2-py2.py3-none-any.whl size=317812365 sha256=7652e33904c14d2112c38b10bc0db0413242b26e0a9d8ae7305980120872a400\n",
            "  Stored in directory: /root/.cache/pip/wheels/34/34/bd/03944534c44b677cd5859f248090daa9fb27b3c8f8e5f49574\n",
            "  Building wheel for intervaltree (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26096 sha256=51519a35387b7c30589e2f305bc24369cc6c5d4af82a00c7a0f6996069791151\n",
            "  Stored in directory: /root/.cache/pip/wheels/fa/80/8c/43488a924a046b733b64de3fac99252674c892a4c3801c0a61\n",
            "Successfully built passim pyspark intervaltree\n",
            "Installing collected packages: nose, pyspark, intervaltree, graphframes, passim\n",
            "Successfully installed graphframes-0.6 intervaltree-3.1.0 nose-1.3.7 passim-2.0.0 pyspark-3.5.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the installation succeeds, you should be able to run the `seriatim` command-line program to see what options are available. (See the documentation for some more detail.) Depending on how you run `pip`, you may need to add Passim's installed location to your `PATH` environment variable."
      ],
      "metadata": {
        "id": "1qw4wrHe6r_D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!seriatim --help"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zWcCEMpWKbLu",
        "outputId": "2db5d4b2-4d36-4e30-d700-ff9d1f038d52"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://repos.spark-packages.org/ added as a remote repository with the name: repo-1\n",
            ":: loading settings :: url = jar:file:/usr/local/lib/python3.10/dist-packages/pyspark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml\n",
            "Ivy Default Cache set to: /root/.ivy2/cache\n",
            "The jars for the packages stored in: /root/.ivy2/jars\n",
            "graphframes#graphframes added as a dependency\n",
            ":: resolving dependencies :: org.apache.spark#spark-submit-parent-991566ae-4e4d-4aa8-886c-34de49830f48;1.0\n",
            "\tconfs: [default]\n",
            "\tfound graphframes#graphframes;0.8.0-spark3.0-s_2.12 in spark-packages\n",
            "\tfound org.slf4j#slf4j-api;1.7.16 in central\n",
            "downloading https://repos.spark-packages.org/graphframes/graphframes/0.8.0-spark3.0-s_2.12/graphframes-0.8.0-spark3.0-s_2.12.jar ...\n",
            "\t[SUCCESSFUL ] graphframes#graphframes;0.8.0-spark3.0-s_2.12!graphframes.jar (133ms)\n",
            "downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar ...\n",
            "\t[SUCCESSFUL ] org.slf4j#slf4j-api;1.7.16!slf4j-api.jar (69ms)\n",
            ":: resolution report :: resolve 2763ms :: artifacts dl 229ms\n",
            "\t:: modules in use:\n",
            "\tgraphframes#graphframes;0.8.0-spark3.0-s_2.12 from spark-packages in [default]\n",
            "\torg.slf4j#slf4j-api;1.7.16 from central in [default]\n",
            "\t---------------------------------------------------------------------\n",
            "\t|                  |            modules            ||   artifacts   |\n",
            "\t|       conf       | number| search|dwnlded|evicted|| number|dwnlded|\n",
            "\t---------------------------------------------------------------------\n",
            "\t|      default     |   2   |   2   |   2   |   0   ||   2   |   2   |\n",
            "\t---------------------------------------------------------------------\n",
            ":: retrieving :: org.apache.spark#spark-submit-parent-991566ae-4e4d-4aa8-886c-34de49830f48\n",
            "\tconfs: [default]\n",
            "\t2 artifacts copied, 0 already retrieved (277kB/18ms)\n",
            "usage: submit-seriatim.py [-h] [-i ID] [-t TEXT] [--locs LOCS] [--pages PAGES] [-l N] [-u N]\n",
            "                          [-m N] [-n N] [--floating-ngrams] [--complete-lines] [-g N]\n",
            "                          [--max-offset N] [--beam N] [--pcopy p] [-a N] [--src-overlap p]\n",
            "                          [--dst-overlap p] [--fields FIELDS [FIELDS ...]] [-f FILTERPAIRS]\n",
            "                          [--all-pairs] [--pairwise] [--docwise] [--linewise] [--to-pairs]\n",
            "                          [--to-extents] [--link-model LINK_MODEL] [--link-features LINK_FEATURES]\n",
            "                          [--log-level {ERROR,WARN,INFO,DEBUG}] [--input-format INPUT_FORMAT]\n",
            "                          [--output-format OUTPUT_FORMAT]\n",
            "                          <path> <path>\n",
            "\n",
            "Passim Alignment\n",
            "\n",
            "positional arguments:\n",
            "  <path>                input data\n",
            "  <path>                output\n",
            "\n",
            "options:\n",
            "  -h, --help            show this help message and exit\n",
            "  -i ID, --id ID        Field for unique document IDs (default: id)\n",
            "  -t TEXT, --text TEXT  Field for document text (default: text)\n",
            "  --locs LOCS           Field for citable loci (default: locs)\n",
            "  --pages PAGES         Field for page location information (default: pages)\n",
            "  -l N, --minDF N       Lower limit on document frequency (default: 2)\n",
            "  -u N, --maxDF N       Upper limit on document frequency (default: 100)\n",
            "  -m N, --min-match N   Minimum number of n-gram matches between documents (default: 5)\n",
            "  -n N, --n N           n-gram order (default: 25)\n",
            "  --floating-ngrams     Allow n-grams to float from word boundaries (default: False)\n",
            "  --complete-lines      Break target alignments at line breaks (default: False)\n",
            "  -g N, --gap N         Minimum size of gap that separates passages (default: 600)\n",
            "  --max-offset N        Maximum offset in global alignment [deprecated] (default: 20)\n",
            "  --beam N              Beam search width (default: 20)\n",
            "  --pcopy p             Probability of copying a character (default: 0.8)\n",
            "  -a N, --min-align N   Minimum length of alignment (default: 50)\n",
            "  --src-overlap p       Source overlap proportion (default: 0.9)\n",
            "  --dst-overlap p       Destination overlap proportion (default: 0.5)\n",
            "  --fields FIELDS [FIELDS ...]\n",
            "                        List of fields to index (default: [])\n",
            "  -f FILTERPAIRS, --filterpairs FILTERPAIRS\n",
            "                        SQL constraint on posting pairs (default: uid < uid2)\n",
            "  --all-pairs           Compute alignments for all pairs. (default: False)\n",
            "  --pairwise            Output pairwise alignments (default: False)\n",
            "  --docwise             Output docwise alignments (default: False)\n",
            "  --linewise            Output linewise alignments (default: False)\n",
            "  --to-pairs            Output pairs and stop (default: False)\n",
            "  --to-extents          Output extents and stop (default: False)\n",
            "  --link-model LINK_MODEL\n",
            "                        Link model in R format (default: None)\n",
            "  --link-features LINK_FEATURES\n",
            "                        Link model features as SQL SELECT (default: None)\n",
            "  --log-level {ERROR,WARN,INFO,DEBUG}\n",
            "                        spark log level (default: WARN)\n",
            "  --input-format INPUT_FORMAT\n",
            "                        Input format (default: json)\n",
            "  --output-format OUTPUT_FORMAT\n",
            "                        Output format (default: json)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's work with some text. First, we load some useful libraries."
      ],
      "metadata": {
        "id": "tgZxMQXm7K4B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from urllib.request import urlopen\n",
        "import sys\n",
        "import passim"
      ],
      "metadata": {
        "id": "v9C__rBlKELk"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll work with the Project Gutenberg transcriptions of two poems by Longfellow."
      ],
      "metadata": {
        "id": "O3DlMQSA7RcQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "childrens_hour = \"\"\"THE CHILDREN'S HOUR\n",
        "\n",
        "Between the dark and the daylight,\n",
        "  When the night is beginning to lower,\n",
        "Comes a pause in the day's occupations,\n",
        " That is known as the Children's Hour.\n",
        "\n",
        "I hear in the chamber above me\n",
        "  The patter of little feet,\n",
        "The sound of a door that is opened,\n",
        "  And voices soft and sweet.\n",
        "\n",
        "From my study I see in the lamplight,\n",
        "  Descending the broad hall stair,\n",
        "Grave Alice, and laughing Allegra,\n",
        "  And Edith with golden hair.\n",
        "\n",
        "A whisper, and then a silence:\n",
        "  Yet I know by their merry eyes\n",
        "They are plotting and planning together\n",
        "  To take me by surprise.\n",
        "\n",
        "A sudden rush from the stairway,\n",
        "  A sudden raid from the hall!\n",
        "By three doors left unguarded\n",
        "  They enter my castle wall!\n",
        "\n",
        "They climb up into my turret\n",
        "  O'er the arms and back of my chair;\n",
        "If I try to escape, they surround me;\n",
        "  They seem to be everywhere.\n",
        "\n",
        "They almost devour me with kisses,\n",
        "  Their arms about me entwine,\n",
        "Till I think of the Bishop of Bingen\n",
        "  In his Mouse-Tower on the Rhine!\n",
        "\n",
        "Do you think, o blue-eyed banditti,\n",
        "  Because you have scaled the wall,\n",
        "Such an old mustache as I am\n",
        "  Is not a match for you all!\n",
        "\n",
        "I have you fast in my fortress,\n",
        "  And will not let you depart,\n",
        "But put you down into the dungeon\n",
        "  In the round-tower of my heart.\n",
        "\n",
        "And there will I keep you forever,\n",
        "  Yes, forever and a day,\n",
        "Till the walls shall crumble to ruin,\n",
        "  And moulder in dust away!\n",
        "\"\"\"\n",
        "\n",
        "reaper = \"\"\"THE REAPER AND THE FLOWERS.\n",
        "\n",
        "There is a Reaper, whose name is Death,\n",
        "  And, with his sickle keen,\n",
        "He reaps the bearded grain at a breath,\n",
        "  And the flowers that grow between.\n",
        "\n",
        "\"Shall I have naught that is fair?\" saith he;\n",
        "  \"Have naught but the bearded grain?\n",
        "Though the breath of these flowers is sweet to me,\n",
        "  I will give them all back again.\"\n",
        "\n",
        "He gazed at the flowers with tearful eyes,\n",
        "  He kissed their drooping leaves;\n",
        "It was for the Lord of Paradise\n",
        "  He bound them in his sheaves.\n",
        "\n",
        "\"My Lord has need of these flowerets gay,\"\n",
        "  The Reaper said, and smiled;\n",
        "\"Dear tokens of the earth are they,\n",
        "  Where he was once a child.\n",
        "\n",
        "\"They shall all bloom in fields of light,\n",
        "  Transplanted by my care,\n",
        "And saints, upon their garments white,\n",
        "  These sacred blossoms wear.\"\n",
        "\n",
        "And the mother gave, in tears and pain,\n",
        "  The flowers she most did love;\n",
        "She knew she should find them all again\n",
        "  In the fields of light above.\n",
        "\n",
        "O, not in cruelty, not in wrath,\n",
        "  The Reaper came that day;\n",
        "'T was an angel visited the green earth,\n",
        "  And took the flowers away.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "HMiFmesWqp3v"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Passim's input consists of one or more _documents_ spread across one or more input files. Whether you define documents as articles, pages, books, or something else will depend on your research question. The minimal document is a record containing a unique identifier in the `id` field and its text in the `text` field. (These field names are configurable.) Other fields will be treated as metadata and passed through to the output unchanged (unless we use the `--field` option discussed below). For now, we assign both of these poems to the group 'gutenberg', which we will use later."
      ],
      "metadata": {
        "id": "-RMgSuA-7lOa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "docs = [{'id': 'childrens_hour', 'group': 'gutenberg', 'text': childrens_hour},\n",
        "        {'id': 'reaper', 'group': 'gutenberg', 'text': reaper}]"
      ],
      "metadata": {
        "id": "mOYRiAvVt65g"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we grab the raw OCR text of two newspaper pages from the Library of Congress's [_Chronicling America_](https://chroniclingamerica.loc.gov/) database and append them to the list of document records."
      ],
      "metadata": {
        "id": "gIfHbbiC8qR5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "id = '/lccn/sn83045462/1860-08-24/ed-1/seq-4'\n",
        "docs.append({'id': id, 'group': '/lccn/sn83045462',\n",
        "             'text': urlopen('https://chroniclingamerica.loc.gov' + id + '/ocr.txt').read().decode('utf-8')})"
      ],
      "metadata": {
        "id": "8gAByg3ayDyt"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "id = '/lccn/sn85025007/1860-08-25/ed-1/seq-1'\n",
        "docs.append({'id': id, 'group': '/lccn/sn85025007',\n",
        "             'text': urlopen('https://chroniclingamerica.loc.gov' + id + '/ocr.txt').read().decode('utf-8')})"
      ],
      "metadata": {
        "id": "SB0THq3P09Sv"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To finish our preparation of the input, we write these four records to a single [JSON Lines](https://jsonlines.org/) file, i.e., one JSON record per line. In general, you can spread your input across as many files and directories as you need, especially in a large project. In addition to JSON, Apache Spark also reads CSV, Parquet, and some other binary formats."
      ],
      "metadata": {
        "id": "yYawUuHR8_1b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('in.json', 'w') as f:\n",
        "  for d in docs:\n",
        "    print(json.dumps(d), file=f)"
      ],
      "metadata": {
        "id": "g2SZpxXaxT6P"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, we run the default text-reuse detection process using the `seriatim` command. (Recent updates to spark make us need to use more memory for this run on the colab virtual machines.)"
      ],
      "metadata": {
        "id": "kX6mEJFb9nzk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Delete old output\n",
        "# !rm -r out_cluster\n",
        "!SPARK_SUBMIT_ARGS=\"--driver-memory 8G --executor-memory 8G\" seriatim in.json out_cluster >& out_cluster.err"
      ],
      "metadata": {
        "id": "hZBefy-B1S01"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To start with, we define a simple function to work with the directories of JSON lines output produced by Spark."
      ],
      "metadata": {
        "id": "eMSPLB0U-JGc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import glob, itertools\n",
        "\n",
        "# Read one JSON record per line\n",
        "def read_jsonl_file(f):\n",
        "  res = []\n",
        "  for line in f:\n",
        "    res.append(json.loads(line))\n",
        "  return res\n",
        "\n",
        "def read_jsonl(d):\n",
        "  return list(itertools.chain.from_iterable([read_jsonl_file(open(f)) for f in glob.glob(d + '/*.json')]))"
      ],
      "metadata": {
        "id": "pujGqy6L2ETG"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "By default, Passim puts the main output in the `out.json` subdirectory. Intermediate files are in `tmp`."
      ],
      "metadata": {
        "id": "qWg0hSvR-R2w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls out_cluster/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xg9-nH6spb3c",
        "outputId": "5af8df0e-74ad-4c32-9113-68ab5bf810d9"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "out.json  tmp\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "When read in all at once, passim's output is a flat array of `dict` records. Each record contains a `cluster` identifier. In this case, the only three output records all share the same cluster ID since _The Children's Hour_ is shared by three of the four input documents. The `text` field contains the reused substring of the original document."
      ],
      "metadata": {
        "id": "hzBwePmf-Z1M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "read_jsonl('out_cluster/out.json')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NrCQu8G4pXSH",
        "outputId": "2c4c717f-1a47-4928-d5ac-aec0604e2680"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'uid': 4069945004964849451,\n",
              "  'cluster': 0,\n",
              "  'begin': 21,\n",
              "  'end': 1346,\n",
              "  'boiler': False,\n",
              "  'src': [{'uid': -7966282239007996852, 'begin': 0, 'end': 1361}],\n",
              "  'size': 3,\n",
              "  'pboiler': 0.0,\n",
              "  'group': '/lccn/sn83045462',\n",
              "  'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'text': \"\\nTHE CHILDREN'S HOUR.\\nBT W. L059V1LL0W.\\nBet-reen the dark ?.<i tae d?yh?ht.\\nWhen the nuht is becinning t > low?r?\\nConn ft p?t)M in thiimf ooeop*'ion?.\\nTaftt n known ft* the Children'* Hoar.\\nI he*- in th* chamber above m?\\n'? ae patt-r of little feet;\\nThe round ot? door thM i* opened.\\nAnd voice* ?of. ? .d sweet.\\nFrom bit stadv I e?e in the UmB ight,\\n^D<>.?seorlin? the brovl ball ?Uir.\\nuiav? Aiion ?nu inurun* a iogi?.\\nAnd Edith with golden hair.\\nA whisper and then a siienc;\\nYet I Know by th- me ry eyet\\nThe> are plot ing and planning togeth\\nTo take me bj surprise.\\nA sudden rush from the stairway,\\nA sadden raid from the hall'\\nBt ?hra~ doors led unguarded.\\nThey outer my castle wall!\\nThey climb up into the Mrret\\nO'er the arms and back of mv chair;\\nIf 1 try to escape they surround me;\\nThey seem to be?everywhar?.\\nThey alrroU devour me with kitsrs.\\nTheir arms about me entwine.\\nTill 1 think of the Bishop ot U ngen\\nIn bis taou.o-towor on the Rhine!\\nDo yon think, O hlu?-e*ed VanditS.\\nB cause >ou have scaled the wall,\\nSuch an old mustaohx as I am\\nla not a ma ten {or you *1 ?\\nI bar* you fa-t in in* fo t'?t?.\\nAnd wnl not let you depart.\\nBut put jou down m.o th dungeon*\\nla the ound to we'of my heart.\\nAnd there will I k<wp you forever,\\nYe?, fo. ever and a da;,\\nTill the vail thill crumble to ruin,\\nAnd moblder in dust away \"},\n",
              " {'uid': 3480939575768595258,\n",
              "  'cluster': 0,\n",
              "  'begin': 48490,\n",
              "  'end': 49802,\n",
              "  'boiler': False,\n",
              "  'src': [{'uid': -7966282239007996852, 'begin': 20, 'end': 1360}],\n",
              "  'size': 3,\n",
              "  'pboiler': 0.0,\n",
              "  'group': '/lccn/sn85025007',\n",
              "  'id': '/lccn/sn85025007/1860-08-25/ed-1/seq-1',\n",
              "  'text': ':\\nBetween the dark and the daylight,\\nWhen the night h beginning to lower,\\nComes a pause in tho day\\'s occupations,\\nThat is known as the Children\\'* llour.\\nI hear in the chamber above me,\\nThe p:ittor of little feet,\\nThe soun-l of a door that is opened,\\nAnd voice# soft and sweet.\\nJr\\'roui inv s.udy 1 see in the lamplight,\\nDrsetU\\'ii.ig the broad hall stair,\\n(Irave Alice and laughing Allegra,\\nAnd Edith with (ioldeu hair.\\nA whisper, and then a silence:\\nYwf L know by their merry eyes\\nThey J.rc plotting and planning together\\nTo take tne by surprise.\\nA sudden rush from the stairway,\\nA sudden raid from the hall!\\nBy three doors lelt unguarded\\nTb\\'.y eiiter iny castle wall!\\nThey climb up into my turret\\nO\\'er the aims and back ot iny chair,\\nin try to escape, they surround mo;\\nThey seem to be everywhere.\\nThey almost devour me with kisses,\\nTheir arms about me entwine,\\nTill 1 think of the Bishop ol\\' Bingen\\nIn his Mouse-Tower on the Rhine !\\nDo you think, 0 blue-eyed banditti,\\nBecause you have scaled the wall,\\nSuch an old muataehe as I am\\nIs uot a match for you all?\\nI have you fast in my fortress,\\nAnd will not let you depart,\\nBut put you down into the dungeons .\\nIn the \"round tower of my heart.\\nAnd there will I keep you forever,\\nYes, forever and a day,\\nTill the walls shall crumble to ruin,\\nAnd moulder in dust awa'},\n",
              " {'uid': -7966282239007996852,\n",
              "  'cluster': 0,\n",
              "  'begin': 0,\n",
              "  'end': 1361,\n",
              "  'boiler': False,\n",
              "  'src': [],\n",
              "  'size': 3,\n",
              "  'pboiler': 0.0,\n",
              "  'group': 'gutenberg',\n",
              "  'id': 'childrens_hour',\n",
              "  'text': \"THE CHILDREN'S HOUR\\n\\nBetween the dark and the daylight,\\n  When the night is beginning to lower,\\nComes a pause in the day's occupations,\\n That is known as the Children's Hour.\\n\\nI hear in the chamber above me\\n  The patter of little feet,\\nThe sound of a door that is opened,\\n  And voices soft and sweet.\\n\\nFrom my study I see in the lamplight,\\n  Descending the broad hall stair,\\nGrave Alice, and laughing Allegra,\\n  And Edith with golden hair.\\n\\nA whisper, and then a silence:\\n  Yet I know by their merry eyes\\nThey are plotting and planning together\\n  To take me by surprise.\\n\\nA sudden rush from the stairway,\\n  A sudden raid from the hall!\\nBy three doors left unguarded\\n  They enter my castle wall!\\n\\nThey climb up into my turret\\n  O'er the arms and back of my chair;\\nIf I try to escape, they surround me;\\n  They seem to be everywhere.\\n\\nThey almost devour me with kisses,\\n  Their arms about me entwine,\\nTill I think of the Bishop of Bingen\\n  In his Mouse-Tower on the Rhine!\\n\\nDo you think, o blue-eyed banditti,\\n  Because you have scaled the wall,\\nSuch an old mustache as I am\\n  Is not a match for you all!\\n\\nI have you fast in my fortress,\\n  And will not let you depart,\\nBut put you down into the dungeon\\n  In the round-tower of my heart.\\n\\nAnd there will I keep you forever,\\n  Yes, forever and a day,\\nTill the walls shall crumble to ruin,\\n  And moulder in dust away!\"}]"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Running Passim in its default mode produces, as we just saw, a set of clusters with shared passages. This mode is useful if we do not know _a priori_ which passages will be resued.\n",
        "\n",
        "If, however, we know something more about the structure of our data, we can tell Passim to consider only certain links between documents. For instance, in our toy corpus of four documents, two of them are clean transcripts of poems and the other two are noisy OCR transcriptions of full newspaper pages.\n",
        "\n",
        "If we want to focus on the line-by-line alignment of certain documents, we can instead get output in `--docwise` mode. For the current toy dataset, we consider only those alignments where the first \"source\" text is in group 'gutenberg' and the second \"target\" text is not. The `--field` and `--filterpairs` options specify these constraints using SQL syntax. When considering a pair of records, Passim keeps the original field names for the source document and appends a `2` to the target document's field names. In this case, we tell Passim to pay attention to the `group` field and to align source documents whose `group` field is 'gutenberg' with target documents whose `group` field is not 'gutenberg'."
      ],
      "metadata": {
        "id": "RY8bSrJa_jJ5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!rm -r out_docwise\n",
        "!seriatim --docwise --field group --filterpairs \"group = 'gutenberg' AND group2 <> 'gutenberg'\" in.json out_docwise >& out_docwise.err\n",
        "\n",
        "# This command produces the same results but would be faster on a large corpus\n",
        "# because it defines a new boolean field `ref` instead of using string comparisons.\n",
        "#!seriatim --docwise --field \"(group = 'gutenberg') AS ref\" --filterpairs 'ref AND !ref2' in.json out_docwise >& out_docwise.err"
      ],
      "metadata": {
        "id": "Yf8gLaz4rxRT",
        "outputId": "02d0f005-6194-486b-8d3e-8723c420f920",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "rm: cannot remove 'out_docwise': No such file or directory\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can use the same simple helper function to get the output of this run."
      ],
      "metadata": {
        "id": "UbbCSsyMBEkW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dw = read_jsonl('out_docwise/out.json')"
      ],
      "metadata": {
        "id": "barN8gJysMnu"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Each output record is now an entire target document broken into lines. The `--filterpairs` option we used ensures that the target documents are those not in group 'gutenberg', i.e., the targets are the OCR'd newspapers. We ignore the other fields and only grab the lines that are aligned to a source document. This is indicated by the presence of the `wits` (i.e., witnesses) field in the line record."
      ],
      "metadata": {
        "id": "3br3BI8VBKKU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "[[line for line in doc['lines'] if 'wits' in line] for doc in dw]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ULWkESfOvFbV",
        "outputId": "8ed2bea7-7a76-48a8-e045-311336bae78d"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[{'begin': 48491,\n",
              "   'text': 'Between the dark and the daylight,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 21,\n",
              "     'alg': 'Between the dark and the daylight,\\n',\n",
              "     'alg2': 'Between the dark and the daylight,\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'Between the dark and the daylight,\\n'}]},\n",
              "  {'begin': 48526,\n",
              "   'text': 'When the night h beginning to lower,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 56,\n",
              "     'alg': '  When the night is beginning to lower,\\n',\n",
              "     'alg2': '--When the night -h beginning to lower,\\n',\n",
              "     'matches': 36,\n",
              "     'text': '  When the night is beginning to lower,\\n'}]},\n",
              "  {'begin': 48563,\n",
              "   'text': \"Comes a pause in tho day's occupations,\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 96,\n",
              "     'alg': \"Comes a pause in the day's occupations,\\n\",\n",
              "     'alg2': \"Comes a pause in tho day's occupations,\\n\",\n",
              "     'matches': 39,\n",
              "     'text': \"Comes a pause in the day's occupations,\\n\"}]},\n",
              "  {'begin': 48603,\n",
              "   'text': \"That is known as the Children'* llour.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 136,\n",
              "     'alg': \" That is known as the Children's -Hour.\\n\\n\",\n",
              "     'alg2': \"-That is known as the Children'* llour.-\\n\",\n",
              "     'matches': 36,\n",
              "     'text': \" That is known as the Children's Hour.\\n\\n\"}]},\n",
              "  {'begin': 48642,\n",
              "   'text': 'I hear in the chamber above me,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 176,\n",
              "     'alg': 'I hear in the chamber above me\\n  ',\n",
              "     'alg2': 'I hear in the chamber above me-,\\n',\n",
              "     'matches': 31,\n",
              "     'text': 'I hear in the chamber above me\\n  '}]},\n",
              "  {'begin': 48674,\n",
              "   'text': 'The p:ittor of little feet,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 209,\n",
              "     'alg': 'The p-atter of little feet,\\n',\n",
              "     'alg2': 'The p:ittor of little feet,\\n',\n",
              "     'matches': 25,\n",
              "     'text': 'The patter of little feet,\\n'}]},\n",
              "  {'begin': 48702,\n",
              "   'text': 'The soun-l of a door that is opened,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 236,\n",
              "     'alg': 'The soun-d of a door that is opened,\\n',\n",
              "     'alg2': 'The soun‐l of a door that is opened,\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'The sound of a door that is opened,\\n'}]},\n",
              "  {'begin': 48739,\n",
              "   'text': 'And voice# soft and sweet.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 272,\n",
              "     'alg': '  And voices soft and sweet.\\n',\n",
              "     'alg2': '--And voice# soft and sweet.\\n',\n",
              "     'matches': 26,\n",
              "     'text': '  And voices soft and sweet.\\n'}]},\n",
              "  {'begin': 48766,\n",
              "   'text': \"Jr'roui inv s.udy 1 see in the lamplight,\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 301,\n",
              "     'alg': '\\n--Fro-m -my study I see in the lamplight,\\n',\n",
              "     'alg2': \"-Jr'roui inv s.udy 1 see in the lamplight,\\n\",\n",
              "     'matches': 32,\n",
              "     'text': '\\nFrom my study I see in the lamplight,\\n'}]},\n",
              "  {'begin': 48808,\n",
              "   'text': \"DrsetU'ii.ig the broad hall stair,\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 340,\n",
              "     'alg': '  Descending the broad hall stair,\\n',\n",
              "     'alg2': \"DrsetU'ii.ig the broad hall stair,\\n\",\n",
              "     'matches': 25,\n",
              "     'text': '  Descending the broad hall stair,\\n'}]},\n",
              "  {'begin': 48843,\n",
              "   'text': '(Irave Alice and laughing Allegra,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 375,\n",
              "     'alg': '-Grave Alice, and laughing Allegra,\\n',\n",
              "     'alg2': '(Irave Alice- and laughing Allegra,\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'Grave Alice, and laughing Allegra,\\n'}]},\n",
              "  {'begin': 48878,\n",
              "   'text': 'And Edith with (ioldeu hair.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 410,\n",
              "     'alg': '  And Edith with -golden hair.\\n',\n",
              "     'alg2': '--And Edith with (ioldeu hair.\\n',\n",
              "     'matches': 26,\n",
              "     'text': '  And Edith with golden hair.\\n'}]},\n",
              "  {'begin': 48907,\n",
              "   'text': 'A whisper, and then a silence:\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 440,\n",
              "     'alg': '\\nA whisper, and then a silence:\\n',\n",
              "     'alg2': '-A whisper, and then a silence:\\n',\n",
              "     'matches': 31,\n",
              "     'text': '\\nA whisper, and then a silence:\\n'}]},\n",
              "  {'begin': 48938,\n",
              "   'text': 'Ywf L know by their merry eyes\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 472,\n",
              "     'alg': '  Yet I know by their merry eyes\\n',\n",
              "     'alg2': '--Ywf L know by their merry eyes\\n',\n",
              "     'matches': 28,\n",
              "     'text': '  Yet I know by their merry eyes\\n'}]},\n",
              "  {'begin': 48969,\n",
              "   'text': 'They J.rc plotting and planning together\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 505,\n",
              "     'alg': 'They -are plotting and planning together\\n',\n",
              "     'alg2': 'They J.rc plotting and planning together\\n',\n",
              "     'matches': 38,\n",
              "     'text': 'They are plotting and planning together\\n'}]},\n",
              "  {'begin': 49010,\n",
              "   'text': 'To take tne by surprise.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 545,\n",
              "     'alg': '  To take -me by surprise.\\n\\n',\n",
              "     'alg2': '--To take tne by surprise.-\\n',\n",
              "     'matches': 23,\n",
              "     'text': '  To take me by surprise.\\n\\n'}]},\n",
              "  {'begin': 49035,\n",
              "   'text': 'A sudden rush from the stairway,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 572,\n",
              "     'alg': 'A sudden rush from the stairway,\\n  ',\n",
              "     'alg2': 'A sudden rush from the stairway,--\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'A sudden rush from the stairway,\\n  '}]},\n",
              "  {'begin': 49068,\n",
              "   'text': 'A sudden raid from the hall!\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 607,\n",
              "     'alg': 'A sudden raid from the hall!\\n',\n",
              "     'alg2': 'A sudden raid from the hall!\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'A sudden raid from the hall!\\n'}]},\n",
              "  {'begin': 49097,\n",
              "   'text': 'By three doors lelt unguarded\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 636,\n",
              "     'alg': 'By three doors left unguarded\\n',\n",
              "     'alg2': 'By three doors lelt unguarded\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'By three doors left unguarded\\n'}]},\n",
              "  {'begin': 49127,\n",
              "   'text': \"Tb'.y eiiter iny castle wall!\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 666,\n",
              "     'alg': '  They e-nter -my castle wall!\\n\\n',\n",
              "     'alg2': \"-Tb'.y eiiter iny castle wall!-\\n\",\n",
              "     'matches': 22,\n",
              "     'text': '  They enter my castle wall!\\n\\n'}]},\n",
              "  {'begin': 49157,\n",
              "   'text': 'They climb up into my turret\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 696,\n",
              "     'alg': 'They climb up into my turret\\n ',\n",
              "     'alg2': 'They climb up into my turret-\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'They climb up into my turret\\n '}]},\n",
              "  {'begin': 49186,\n",
              "   'text': \"O'er the aims and back ot iny chair,\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 726,\n",
              "     'alg': \" O'er the arms and back of -my chair;\\n\",\n",
              "     'alg2': \"-O'er the aims and back ot iny chair,\\n\",\n",
              "     'matches': 32,\n",
              "     'text': \" O'er the arms and back of my chair;\\n\"}]},\n",
              "  {'begin': 49223,\n",
              "   'text': 'in try to escape, they surround mo;\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 763,\n",
              "     'alg': 'If I try to escape, they surround me;\\n',\n",
              "     'alg2': '--in try to escape, they surround mo;\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'If I try to escape, they surround me;\\n'}]},\n",
              "  {'begin': 49259,\n",
              "   'text': 'They seem to be everywhere.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 801,\n",
              "     'alg': '  They seem to be everywhere.\\n\\n',\n",
              "     'alg2': '--They seem to be everywhere.-\\n',\n",
              "     'matches': 28,\n",
              "     'text': '  They seem to be everywhere.\\n\\n'}]},\n",
              "  {'begin': 49287,\n",
              "   'text': 'They almost devour me with kisses,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 832,\n",
              "     'alg': 'They almost devour me with kisses,\\n  ',\n",
              "     'alg2': 'They almost devour me with kisses,--\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'They almost devour me with kisses,\\n  '}]},\n",
              "  {'begin': 49322,\n",
              "   'text': 'Their arms about me entwine,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 869,\n",
              "     'alg': 'Their arms about me entwine,\\n',\n",
              "     'alg2': 'Their arms about me entwine,\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'Their arms about me entwine,\\n'}]},\n",
              "  {'begin': 49351,\n",
              "   'text': \"Till 1 think of the Bishop ol' Bingen\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 898,\n",
              "     'alg': 'Till I think of the Bishop o-f Bingen\\n  ',\n",
              "     'alg2': \"Till 1 think of the Bishop ol' Bingen--\\n\",\n",
              "     'matches': 35,\n",
              "     'text': 'Till I think of the Bishop of Bingen\\n  '}]},\n",
              "  {'begin': 49389,\n",
              "   'text': 'In his Mouse-Tower on the Rhine !\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 937,\n",
              "     'alg': 'In his Mouse‐Tower on the Rhine!\\n-\\n',\n",
              "     'alg2': 'In his Mouse‐Tower on the Rhine- !\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'In his Mouse‐Tower on the Rhine!\\n\\n'}]},\n",
              "  {'begin': 49423,\n",
              "   'text': 'Do you think, 0 blue-eyed banditti,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 971,\n",
              "     'alg': 'Do you think, o blue‐eyed banditti,\\n  ',\n",
              "     'alg2': 'Do you think, 0 blue‐eyed banditti,--\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'Do you think, o blue‐eyed banditti,\\n  '}]},\n",
              "  {'begin': 49459,\n",
              "   'text': 'Because you have scaled the wall,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1009,\n",
              "     'alg': 'Because you have scaled the wall,\\n',\n",
              "     'alg2': 'Because you have scaled the wall,\\n',\n",
              "     'matches': 34,\n",
              "     'text': 'Because you have scaled the wall,\\n'}]},\n",
              "  {'begin': 49493,\n",
              "   'text': 'Such an old muataehe as I am\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1043,\n",
              "     'alg': 'Such an old mustache as I am\\n',\n",
              "     'alg2': 'Such an old muataehe as I am\\n',\n",
              "     'matches': 27,\n",
              "     'text': 'Such an old mustache as I am\\n'}]},\n",
              "  {'begin': 49522,\n",
              "   'text': 'Is uot a match for you all?\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1072,\n",
              "     'alg': '  Is not a match for you all!\\n\\n',\n",
              "     'alg2': '--Is uot a match for you all?-\\n',\n",
              "     'matches': 26,\n",
              "     'text': '  Is not a match for you all!\\n\\n'}]},\n",
              "  {'begin': 49550,\n",
              "   'text': 'I have you fast in my fortress,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1103,\n",
              "     'alg': 'I have you fast in my fortress,\\n  ',\n",
              "     'alg2': 'I have you fast in my fortress,--\\n',\n",
              "     'matches': 32,\n",
              "     'text': 'I have you fast in my fortress,\\n  '}]},\n",
              "  {'begin': 49582,\n",
              "   'text': 'And will not let you depart,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1137,\n",
              "     'alg': 'And will not let you depart,\\n',\n",
              "     'alg2': 'And will not let you depart,\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'And will not let you depart,\\n'}]},\n",
              "  {'begin': 49611,\n",
              "   'text': 'But put you down into the dungeons .\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1166,\n",
              "     'alg': 'But put you down into the dungeon-\\n  ',\n",
              "     'alg2': 'But put you down into the dungeons .\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'But put you down into the dungeon\\n  '}]},\n",
              "  {'begin': 49648,\n",
              "   'text': 'In the \"round tower of my heart.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1202,\n",
              "     'alg': 'In the -round‐tower of my heart.\\n\\n',\n",
              "     'alg2': 'In the \"round tower of my heart.-\\n',\n",
              "     'matches': 31,\n",
              "     'text': 'In the round‐tower of my heart.\\n\\n'}]},\n",
              "  {'begin': 49681,\n",
              "   'text': 'And there will I keep you forever,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1235,\n",
              "     'alg': 'And there will I keep you forever,\\n  ',\n",
              "     'alg2': 'And there will I keep you forever,--\\n',\n",
              "     'matches': 35,\n",
              "     'text': 'And there will I keep you forever,\\n  '}]},\n",
              "  {'begin': 49716,\n",
              "   'text': 'Yes, forever and a day,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1272,\n",
              "     'alg': 'Yes, forever and a day,\\n',\n",
              "     'alg2': 'Yes, forever and a day,\\n',\n",
              "     'matches': 24,\n",
              "     'text': 'Yes, forever and a day,\\n'}]},\n",
              "  {'begin': 49740,\n",
              "   'text': 'Till the walls shall crumble to ruin,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1296,\n",
              "     'alg': 'Till the walls shall crumble to ruin,\\n  ',\n",
              "     'alg2': 'Till the walls shall crumble to ruin,--\\n',\n",
              "     'matches': 38,\n",
              "     'text': 'Till the walls shall crumble to ruin,\\n  '}]},\n",
              "  {'begin': 49778,\n",
              "   'text': 'And moulder in dust away.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1336,\n",
              "     'alg': 'And moulder in dust away',\n",
              "     'alg2': 'And moulder in dust away',\n",
              "     'matches': 24,\n",
              "     'text': 'And moulder in dust away'}]}],\n",
              " [{'begin': 21,\n",
              "   'text': \"THE CHILDREN'S HOUR.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 0,\n",
              "     'alg': \"THE CHILDREN'S HOUR-\\n\",\n",
              "     'alg2': \"THE CHILDREN'S HOUR.\\n\",\n",
              "     'matches': 20,\n",
              "     'text': \"THE CHILDREN'S HOUR\\n\"}]},\n",
              "  {'begin': 42,\n",
              "   'text': 'BT W. L059V1LL0W.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 20,\n",
              "     'alg': '--\\n---------------',\n",
              "     'alg2': 'BT W. L059V1LL0W.\\n',\n",
              "     'matches': 1,\n",
              "     'text': '\\n'}]},\n",
              "  {'begin': 60,\n",
              "   'text': 'Bet-reen the dark ?.<i tae d?yh?ht.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 21,\n",
              "     'alg': 'Bet-ween the dark -and the daylight,\\n  ',\n",
              "     'alg2': 'Bet‐reen the dark ?.<i tae d?y-h?ht--.\\n',\n",
              "     'matches': 25,\n",
              "     'text': 'Between the dark and the daylight,\\n  '}]},\n",
              "  {'begin': 96,\n",
              "   'text': 'When the nuht is becinning t > low?r?\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 58,\n",
              "     'alg': 'When the night is beginning t-o lower,\\n',\n",
              "     'alg2': 'When the n-uht is becinning t > low?r?\\n',\n",
              "     'matches': 32,\n",
              "     'text': 'When the night is beginning to lower,\\n'}]},\n",
              "  {'begin': 134,\n",
              "   'text': \"Conn ft p?t)M in thiimf ooeop*'ion?.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 96,\n",
              "     'alg': \"Comes a pause in the day's occupations,\\n\",\n",
              "     'alg2': \"Conn ft p?t)M in th---iimf ooeop*'ion?.\\n\",\n",
              "     'matches': 17,\n",
              "     'text': \"Comes a pause in the day's occupations,\\n\"}]},\n",
              "  {'begin': 171,\n",
              "   'text': \"Taftt n known ft* the Children'* Hoar.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 136,\n",
              "     'alg': \" That is known -as the Children's Hour.\\n\",\n",
              "     'alg2': \"-Taftt n known ft* the Children'* Hoar.\\n\",\n",
              "     'matches': 29,\n",
              "     'text': \" That is known as the Children's Hour.\\n\"}]},\n",
              "  {'begin': 210,\n",
              "   'text': 'I he*- in th* chamber above m?\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 175,\n",
              "     'alg': '\\nI hear in the chamber above me\\n',\n",
              "     'alg2': '-I he*‐ in th* chamber above m?\\n',\n",
              "     'matches': 27,\n",
              "     'text': '\\nI hear in the chamber above me\\n'}]},\n",
              "  {'begin': 241,\n",
              "   'text': \"'? ae patt-r of little feet;\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 207,\n",
              "     'alg': '  The patter of little feet,\\n',\n",
              "     'alg2': \"'? ae patt‐r of little feet;\\n\",\n",
              "     'matches': 23,\n",
              "     'text': '  The patter of little feet,\\n'}]},\n",
              "  {'begin': 270,\n",
              "   'text': 'The round ot? door thM i* opened.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 236,\n",
              "     'alg': 'The sound of a door that is opened,\\n  ',\n",
              "     'alg2': 'The round o-t? door th-M i* opened--.\\n',\n",
              "     'matches': 28,\n",
              "     'text': 'The sound of a door that is opened,\\n  '}]},\n",
              "  {'begin': 304,\n",
              "   'text': 'And voice* ?of. ? .d sweet.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 274,\n",
              "     'alg': 'And voices soft -and sweet.\\n',\n",
              "     'alg2': 'And voice* ?of. ? .d sweet.\\n',\n",
              "     'matches': 22,\n",
              "     'text': 'And voices soft and sweet.\\n'}]},\n",
              "  {'begin': 332,\n",
              "   'text': 'From bit stadv I e?e in the UmB ight,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 301,\n",
              "     'alg': '\\nFrom -my study I see in the lamplight,\\n',\n",
              "     'alg2': '-From bit stadv I e?e in the -UmB ight,\\n',\n",
              "     'matches': 28,\n",
              "     'text': '\\nFrom my study I see in the lamplight,\\n'}]},\n",
              "  {'begin': 370,\n",
              "   'text': '^D<>.?seorlin? the brovl ball ?Uir.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 340,\n",
              "     'alg': '  --Descending the broad hall stair,\\n',\n",
              "     'alg2': '^D<>.?seorlin? the brovl ball -?Uir.\\n',\n",
              "     'matches': 19,\n",
              "     'text': '  Descending the broad hall stair,\\n'}]},\n",
              "  {'begin': 406,\n",
              "   'text': 'uiav? Aiion ?nu inurun* a iogi?.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 375,\n",
              "     'alg': 'Grave Alice, and laughing Allegra,\\n',\n",
              "     'alg2': 'uiav? A-iion ?nu inu-run* a iogi?.\\n',\n",
              "     'matches': 14,\n",
              "     'text': 'Grave Alice, and laughing Allegra,\\n'}]},\n",
              "  {'begin': 439,\n",
              "   'text': 'And Edith with golden hair.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 410,\n",
              "     'alg': '  And Edith with golden hair.\\n',\n",
              "     'alg2': '--And Edith with golden hair.\\n',\n",
              "     'matches': 28,\n",
              "     'text': '  And Edith with golden hair.\\n'}]},\n",
              "  {'begin': 467,\n",
              "   'text': 'A whisper and then a siienc;\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 440,\n",
              "     'alg': '\\nA whisper, and then a silence:\\n  ',\n",
              "     'alg2': '-A whisper- and then a siienc---;\\n',\n",
              "     'matches': 27,\n",
              "     'text': '\\nA whisper, and then a silence:\\n  '}]},\n",
              "  {'begin': 496,\n",
              "   'text': 'Yet I Know by th- me ry eyet\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 474,\n",
              "     'alg': 'Yet I know by their merry eyes\\n',\n",
              "     'alg2': 'Yet I Know by th--‐ me ry eyet\\n',\n",
              "     'matches': 26,\n",
              "     'text': 'Yet I know by their merry eyes\\n'}]},\n",
              "  {'begin': 525,\n",
              "   'text': 'The> are plot ing and planning togeth\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 505,\n",
              "     'alg': 'They are plotting and planning together\\n',\n",
              "     'alg2': 'The> are plot ing and planning togeth--\\n',\n",
              "     'matches': 36,\n",
              "     'text': 'They are plotting and planning together\\n'}]},\n",
              "  {'begin': 563,\n",
              "   'text': 'To take me bj surprise.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 545,\n",
              "     'alg': '  To take me by surprise.\\n\\n',\n",
              "     'alg2': '--To take me bj surprise.-\\n',\n",
              "     'matches': 23,\n",
              "     'text': '  To take me by surprise.\\n\\n'}]},\n",
              "  {'begin': 587,\n",
              "   'text': 'A sudden rush from the stairway,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 572,\n",
              "     'alg': 'A sudden rush from the stairway,\\n',\n",
              "     'alg2': 'A sudden rush from the stairway,\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'A sudden rush from the stairway,\\n'}]},\n",
              "  {'begin': 620,\n",
              "   'text': \"A sadden raid from the hall'\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 605,\n",
              "     'alg': '  A sudden raid from the hall!\\n',\n",
              "     'alg2': \"--A sadden raid from the hall'\\n\",\n",
              "     'matches': 27,\n",
              "     'text': '  A sudden raid from the hall!\\n'}]},\n",
              "  {'begin': 649,\n",
              "   'text': 'Bt ?hra~ doors led unguarded.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 636,\n",
              "     'alg': 'By three doors left unguarded\\n  ',\n",
              "     'alg2': 'Bt ?hra~ doors le-d unguarded-.\\n',\n",
              "     'matches': 24,\n",
              "     'text': 'By three doors left unguarded\\n  '}]},\n",
              "  {'begin': 679,\n",
              "   'text': 'They outer my castle wall!\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 668,\n",
              "     'alg': 'They enter my castle wall!\\n',\n",
              "     'alg2': 'They outer my castle wall!\\n',\n",
              "     'matches': 25,\n",
              "     'text': 'They enter my castle wall!\\n'}]},\n",
              "  {'begin': 706,\n",
              "   'text': 'They climb up into the Mrret\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 695,\n",
              "     'alg': '\\nThey climb up into my turret\\n',\n",
              "     'alg2': '-They climb up into the Mrret\\n',\n",
              "     'matches': 24,\n",
              "     'text': '\\nThey climb up into my turret\\n'}]},\n",
              "  {'begin': 735,\n",
              "   'text': \"O'er the arms and back of mv chair;\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 725,\n",
              "     'alg': \"  O'er the arms and back of my chair;\\n\",\n",
              "     'alg2': \"--O'er the arms and back of mv chair;\\n\",\n",
              "     'matches': 35,\n",
              "     'text': \"  O'er the arms and back of my chair;\\n\"}]},\n",
              "  {'begin': 771,\n",
              "   'text': 'If 1 try to escape they surround me;\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 763,\n",
              "     'alg': 'If I try to escape, they surround me;\\n',\n",
              "     'alg2': 'If 1 try to escape- they surround me;\\n',\n",
              "     'matches': 36,\n",
              "     'text': 'If I try to escape, they surround me;\\n'}]},\n",
              "  {'begin': 808,\n",
              "   'text': 'They seem to be?everywhar?.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 801,\n",
              "     'alg': '  They seem to be everywhere.\\n',\n",
              "     'alg2': '--They seem to be?everywhar?.\\n',\n",
              "     'matches': 25,\n",
              "     'text': '  They seem to be everywhere.\\n'}]},\n",
              "  {'begin': 836,\n",
              "   'text': 'They alrroU devour me with kitsrs.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 831,\n",
              "     'alg': '\\nThey almost devour me with kisses,\\n',\n",
              "     'alg2': '-They alrroU devour me with kitsrs.\\n',\n",
              "     'matches': 28,\n",
              "     'text': '\\nThey almost devour me with kisses,\\n'}]},\n",
              "  {'begin': 871,\n",
              "   'text': 'Their arms about me entwine.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 867,\n",
              "     'alg': '  Their arms about me entwine,\\n',\n",
              "     'alg2': '--Their arms about me entwine.\\n',\n",
              "     'matches': 28,\n",
              "     'text': '  Their arms about me entwine,\\n'}]},\n",
              "  {'begin': 900,\n",
              "   'text': 'Till 1 think of the Bishop ot U ngen\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 898,\n",
              "     'alg': 'Till I think of the Bishop of Bingen\\n',\n",
              "     'alg2': 'Till 1 think of the Bishop ot U ngen\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'Till I think of the Bishop of Bingen\\n'}]},\n",
              "  {'begin': 937,\n",
              "   'text': 'In bis taou.o-towor on the Rhine!\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 935,\n",
              "     'alg': '  In his -Mouse‐Tower on the Rhine!\\n',\n",
              "     'alg2': '--In bis taou.o‐towor on the Rhine!\\n',\n",
              "     'matches': 28,\n",
              "     'text': '  In his Mouse‐Tower on the Rhine!\\n'}]},\n",
              "  {'begin': 971,\n",
              "   'text': 'Do yon think, O hlu?-e*ed VanditS.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 970,\n",
              "     'alg': '\\nDo you think, o blue‐eyed banditti,\\n  ',\n",
              "     'alg2': '-Do yon think, O hlu?‐e*ed Vandi-t-S-.\\n',\n",
              "     'matches': 28,\n",
              "     'text': '\\nDo you think, o blue‐eyed banditti,\\n  '}]},\n",
              "  {'begin': 1006,\n",
              "   'text': 'B cause >ou have scaled the wall,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1009,\n",
              "     'alg': 'Because you have scaled the wall,\\n',\n",
              "     'alg2': 'B cause >ou have scaled the wall,\\n',\n",
              "     'matches': 32,\n",
              "     'text': 'Because you have scaled the wall,\\n'}]},\n",
              "  {'begin': 1040,\n",
              "   'text': 'Such an old mustaohx as I am\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1043,\n",
              "     'alg': 'Such an old mustache as I am\\n',\n",
              "     'alg2': 'Such an old mustaohx as I am\\n',\n",
              "     'matches': 27,\n",
              "     'text': 'Such an old mustache as I am\\n'}]},\n",
              "  {'begin': 1069,\n",
              "   'text': 'la not a ma ten {or you *1 ?\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1072,\n",
              "     'alg': '  Is not a ma-tch for you all!\\n',\n",
              "     'alg2': '--la not a ma ten {or you *1 ?\\n',\n",
              "     'matches': 19,\n",
              "     'text': '  Is not a match for you all!\\n'}]},\n",
              "  {'begin': 1098,\n",
              "   'text': \"I bar* you fa-t in in* fo t'?t?.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1102,\n",
              "     'alg': '\\nI have you fast in -my fortress,\\n  ',\n",
              "     'alg2': \"-I bar* you fa‐t in in* fo t'-?t?-.\\n\",\n",
              "     'matches': 20,\n",
              "     'text': '\\nI have you fast in my fortress,\\n  '}]},\n",
              "  {'begin': 1131,\n",
              "   'text': 'And wnl not let you depart.\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1137,\n",
              "     'alg': 'And will not let you depart,\\n',\n",
              "     'alg2': 'And w-nl not let you depart.\\n',\n",
              "     'matches': 26,\n",
              "     'text': 'And will not let you depart,\\n'}]},\n",
              "  {'begin': 1159,\n",
              "   'text': 'But put jou down m.o th dungeon*\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1166,\n",
              "     'alg': 'But put you down into the dungeon\\n  ',\n",
              "     'alg2': 'But put jou down -m.o th- dungeon-*\\n',\n",
              "     'matches': 29,\n",
              "     'text': 'But put you down into the dungeon\\n  '}]},\n",
              "  {'begin': 1192,\n",
              "   'text': \"la the ound to we'of my heart.\\n\",\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1202,\n",
              "     'alg': 'In the round‐to-wer of my heart.\\n',\n",
              "     'alg2': \"la the -ound to we'-of my heart.\\n\",\n",
              "     'matches': 26,\n",
              "     'text': 'In the round‐tower of my heart.\\n'}]},\n",
              "  {'begin': 1223,\n",
              "   'text': 'And there will I k<wp you forever,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1234,\n",
              "     'alg': '\\nAnd there will I keep you forever,\\n',\n",
              "     'alg2': '-And there will I k<wp you forever,\\n',\n",
              "     'matches': 33,\n",
              "     'text': '\\nAnd there will I keep you forever,\\n'}]},\n",
              "  {'begin': 1258,\n",
              "   'text': 'Ye?, fo. ever and a da;,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1270,\n",
              "     'alg': '  Yes, fo-rever and a day,\\n',\n",
              "     'alg2': '--Ye?, fo. ever and a da;,\\n',\n",
              "     'matches': 21,\n",
              "     'text': '  Yes, forever and a day,\\n'}]},\n",
              "  {'begin': 1283,\n",
              "   'text': 'Till the vail thill crumble to ruin,\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1296,\n",
              "     'alg': 'Till the walls shall crumble to ruin,\\n',\n",
              "     'alg2': 'Till the vail- thill crumble to ruin,\\n',\n",
              "     'matches': 33,\n",
              "     'text': 'Till the walls shall crumble to ruin,\\n'}]},\n",
              "  {'begin': 1320,\n",
              "   'text': 'And moblder in dust away !\\n',\n",
              "   'wits': [{'group': 'gutenberg',\n",
              "     'id': 'childrens_hour',\n",
              "     'begin': 1334,\n",
              "     'alg': '  And moulder in dust away-!',\n",
              "     'alg2': '--And moblder in dust away !',\n",
              "     'matches': 24,\n",
              "     'text': '  And moulder in dust away!'}]}]]"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For the small dataset in this demo, it's easy enough to read one file at a time. To work with bigger datasets spread over multiple files, it is more convenient to use Apache Spark itself."
      ],
      "metadata": {
        "id": "ebWA9BpLBqKb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql import SparkSession, Row, DataFrame\n",
        "from pyspark.sql.functions import (col, explode)\n",
        "import pyspark.sql.functions as f\n",
        "spark = SparkSession.builder.appName('Passim Demo').getOrCreate()"
      ],
      "metadata": {
        "id": "_vCJ6bf9vk9r"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We read in the JSON output from the docwise Passim run."
      ],
      "metadata": {
        "id": "HxoRuE9_CNxZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dwdata = spark.read.json('out_docwise/out.json')"
      ],
      "metadata": {
        "id": "dd8NTri-xbQl"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Spark infers the schema of this data, so we can see the available fields and their types."
      ],
      "metadata": {
        "id": "yFapJLuyCTvq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dwdata.printSchema()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q_qWY_k_y4Oj",
        "outputId": "da7d89e3-691f-4aa6-f187-3c898a14e1a2"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- group: string (nullable = true)\n",
            " |-- id: string (nullable = true)\n",
            " |-- lines: array (nullable = true)\n",
            " |    |-- element: struct (containsNull = true)\n",
            " |    |    |-- begin: long (nullable = true)\n",
            " |    |    |-- text: string (nullable = true)\n",
            " |    |    |-- wits: array (nullable = true)\n",
            " |    |    |    |-- element: struct (containsNull = true)\n",
            " |    |    |    |    |-- alg: string (nullable = true)\n",
            " |    |    |    |    |-- alg2: string (nullable = true)\n",
            " |    |    |    |    |-- begin: long (nullable = true)\n",
            " |    |    |    |    |-- group: string (nullable = true)\n",
            " |    |    |    |    |-- id: string (nullable = true)\n",
            " |    |    |    |    |-- matches: long (nullable = true)\n",
            " |    |    |    |    |-- text: string (nullable = true)\n",
            " |-- text: string (nullable = true)\n",
            " |-- uid: long (nullable = true)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As above, we focus on those lines that are aligned to a reference text. The functions in the Spark python API correspond to operations in relational algebra such as select, filter, etc. (This style of working with data frames may be familiar to users of Pandas in python or dplyr in R.)"
      ],
      "metadata": {
        "id": "maVtwoPGCgiC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lines = dwdata.select('id', f.explode('lines').alias('line')\n",
        "  ).filter(col('line')['wits'].isNotNull()\n",
        "  ).select('id', col('line.begin'),\n",
        "           col('line.wits')[0]['alg'].alias('ref'),\n",
        "           col('line.wits')[0]['alg2'].alias('ocr')\n",
        "  ).sort('id', 'begin')\n",
        "[r.asDict(True) for r in lines.limit(20).collect()]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ScLNvbqcxi6N",
        "outputId": "5374eeb4-01cc-486e-fb7c-2aa76fc78e22"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 21,\n",
              "  'ref': \"THE CHILDREN'S HOUR-\\n\",\n",
              "  'ocr': \"THE CHILDREN'S HOUR.\\n\"},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 42,\n",
              "  'ref': '--\\n---------------',\n",
              "  'ocr': 'BT W. L059V1LL0W.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 60,\n",
              "  'ref': 'Bet-ween the dark -and the daylight,\\n  ',\n",
              "  'ocr': 'Bet‐reen the dark ?.<i tae d?y-h?ht--.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 96,\n",
              "  'ref': 'When the night is beginning t-o lower,\\n',\n",
              "  'ocr': 'When the n-uht is becinning t > low?r?\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 134,\n",
              "  'ref': \"Comes a pause in the day's occupations,\\n\",\n",
              "  'ocr': \"Conn ft p?t)M in th---iimf ooeop*'ion?.\\n\"},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 171,\n",
              "  'ref': \" That is known -as the Children's Hour.\\n\",\n",
              "  'ocr': \"-Taftt n known ft* the Children'* Hoar.\\n\"},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 210,\n",
              "  'ref': '\\nI hear in the chamber above me\\n',\n",
              "  'ocr': '-I he*‐ in th* chamber above m?\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 241,\n",
              "  'ref': '  The patter of little feet,\\n',\n",
              "  'ocr': \"'? ae patt‐r of little feet;\\n\"},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 270,\n",
              "  'ref': 'The sound of a door that is opened,\\n  ',\n",
              "  'ocr': 'The round o-t? door th-M i* opened--.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 304,\n",
              "  'ref': 'And voices soft -and sweet.\\n',\n",
              "  'ocr': 'And voice* ?of. ? .d sweet.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 332,\n",
              "  'ref': '\\nFrom -my study I see in the lamplight,\\n',\n",
              "  'ocr': '-From bit stadv I e?e in the -UmB ight,\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 370,\n",
              "  'ref': '  --Descending the broad hall stair,\\n',\n",
              "  'ocr': '^D<>.?seorlin? the brovl ball -?Uir.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 406,\n",
              "  'ref': 'Grave Alice, and laughing Allegra,\\n',\n",
              "  'ocr': 'uiav? A-iion ?nu inu-run* a iogi?.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 439,\n",
              "  'ref': '  And Edith with golden hair.\\n',\n",
              "  'ocr': '--And Edith with golden hair.\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 467,\n",
              "  'ref': '\\nA whisper, and then a silence:\\n  ',\n",
              "  'ocr': '-A whisper- and then a siienc---;\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 496,\n",
              "  'ref': 'Yet I know by their merry eyes\\n',\n",
              "  'ocr': 'Yet I Know by th--‐ me ry eyet\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 525,\n",
              "  'ref': 'They are plotting and planning together\\n',\n",
              "  'ocr': 'The> are plot ing and planning togeth--\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 563,\n",
              "  'ref': '  To take me by surprise.\\n\\n',\n",
              "  'ocr': '--To take me bj surprise.-\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 587,\n",
              "  'ref': 'A sudden rush from the stairway,\\n',\n",
              "  'ocr': 'A sudden rush from the stairway,\\n'},\n",
              " {'id': '/lccn/sn83045462/1860-08-24/ed-1/seq-4',\n",
              "  'begin': 620,\n",
              "  'ref': '  A sudden raid from the hall!\\n',\n",
              "  'ocr': \"--A sadden raid from the hall'\\n\"}]"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    }
  ]
}