tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "130694eb",
   "metadata": {},
   "source": [
    "# Analiticcl tutorial (using Python)\n",
    "\n",
    "Analiticcl is an approximate string matching or fuzzy-matching system that can be used for spelling\n",
    "correction or text normalisation (such as post-OCR correction or post-HTR correction). Texts can be checked against a\n",
    "validated or corpus-derived lexicon (with or without frequency information) and spelling variants will be returned.\n",
    "\n",
    "To understand the theoretical background behind analiticcl, we recommend you to also view [this presentation\n",
    "video](https://diode.zone/w/kkrqA4MocGwxyC3s68Zsq7) that was presented at the KNAW Humanities Cluster in January 2022.\n",
    "\n",
    "Analiticcl can be invoked from either the command-line or via Python using the binding.\n",
    "In this tutorial, we will use the latter option and explore some of the functionality of analiticcl.\n",
    "\n",
    "## Installation\n",
    "\n",
    "First of all, we need to install analiticcl, in a Jupyter Notebook this is simply accomplished as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fa12b213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Collecting analiticcl\n",
      "  Downloading analiticcl-0.4.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: analiticcl\n",
      "Successfully installed analiticcl-0.4.4\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install analiticcl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64c407b",
   "metadata": {},
   "source": [
    "When invoked from the command line instead, do the following to create a Python virtual environment and install analiticcl in it:\n",
    "\n",
    "```\n",
    "$ python -m venv env\n",
    "$ . env/bin/activate\n",
    "$ pip install analiticcl\n",
    "```\n",
    "\n",
    "Now analiticcl is installed, we can import the module. As we usually only need three main classes, we import only these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "44323995",
   "metadata": {},
   "outputs": [],
   "source": [
    "from analiticcl import VariantModel, Weights, SearchParameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191592cc",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "Analiticcl doesn't do much out-of-the-box and is only as good as the data you feed it. It is a fairly low-level tool that is quite versatile, and it's up to you to wield it effectively. It specifically needs *lexicons* or *variant lists* to operate, these contain the words or phrases that the system will match against.\n",
    "\n",
    "**Advanced note:** All input for analiticcl must be UTF-8 encoded and use unix-style line endings, NFC unicode normalisation is strongly recommended.\n",
    "\n",
    "### Alphabet file\n",
    "\n",
    "We first of all need an *alphabet file* which simply defines all characters in the alphabet, grouping certain character variants together if desired. See the [README.md](README.md) for further documentation on this. We simply take the example alphabet file that is supplied with analiticcl. The alphabet file is a TSV file (tab separated fields) containing all characters of the alphabet. Each line describes a\n",
    "single alphabet 'character', all columns on the same line are considered equivalent variants of the same character from the perspective of analiticcl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ce3b4109",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\tA\tá\tà\tÁ\tÀ\tä\tÄ\tã\tÃ\tâ\tÂ\n",
      "e\tE\të\té\tè\tê\tË\tÉ\tÈ\tÊ\tæ\tÆ\n",
      "o\tO\tö\tó\tò\tõ\tô\tÖ\tÓ\tÒ\tÕ\tÔ\tå\tÅ\tø\tœ\u001b\n",
      "i\tI\tï\tí\tÍ\n",
      "u\tU\tú\tÚ\tü\tÜ\n",
      "y\tY\n",
      "b\tB\n",
      "c\tC\n",
      "d\tD\n",
      "f\tf\n",
      "g\tG\n",
      "h\tH\n",
      "k\tk\n",
      "l\tL\n",
      "m\tM\n",
      "n\tN\tñ\tÑ\n",
      "p\tP\n",
      "r\tR\n",
      "s\tS\n",
      "t\tT\n",
      "j\tJ\n",
      "v\tV\n",
      "w\tW\n",
      "q\tQ\n",
      "x\tX\n",
      "z\tZ\n",
      "\"\t``\t''\n",
      "'\n",
      "\\s\t\\t\n",
      ".\t,\t:\t?\t!\n",
      "0\t1\t2\t3\t4\t5\t6\t7\t8\t9\n",
      "\n"
     ]
    }
   ],
   "source": [
    "alphabet_file = \"examples/simple.alphabet.tsv\"\n",
    "\n",
    "with open(alphabet_file,'r', encoding='utf-8') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9664e0",
   "metadata": {},
   "source": [
    "### Lexicon\n",
    "\n",
    "In this tutorial we will use an English lexicon from the [GNU aspell](http://aspell.net/) project, a commonly used spell checker library. It simply contains one word per line. An example is supplied with analiticcl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "281a84a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "lexicon_file = \"examples/eng.aspell.lexicon\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "589888ae",
   "metadata": {},
   "source": [
    "## Variant Model\n",
    "\n",
    "### Building\n",
    "\n",
    "We now have all we need to build our first variant model using Analiticcl.  A variant model enables quickly and efficiently matching any input to specified lexicons, effectively matching the input text against the lexicons and in doing so finding variants of the input (or variants of the lexicon entries, it's only a matter of perspective)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "02ff266a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = VariantModel(alphabet_file, Weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30211f8c",
   "metadata": {},
   "source": [
    "For the time being we're content with the default weights (more about these later), passed as second parameter.\n",
    "\n",
    "The model is still empty upon instantiation. We need to feed it with one or more lexicons. Let's pass the English aspell lexicon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "969492eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.read_lexicon(lexicon_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d9cc3e3",
   "metadata": {},
   "source": [
    "After loading all lexicon, we build the model as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9047d95a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing anagram values for all items in the lexicon...\n",
      " - Found 119773 instances\n",
      "Adding all instances to the index...\n",
      " - Found 108802 anagrams\n",
      "Creating sorted secondary index...\n",
      "Sorting secondary index...\n",
      " - Found 27 anagrams of length 1\n",
      " - Found 248 anagrams of length 2\n",
      " - Found 942 anagrams of length 3\n",
      " - Found 2593 anagrams of length 4\n",
      " - Found 5623 anagrams of length 5\n",
      " - Found 10163 anagrams of length 6\n",
      " - Found 14617 anagrams of length 7\n",
      " - Found 16911 anagrams of length 8\n",
      " - Found 16391 anagrams of length 9\n",
      " - Found 13930 anagrams of length 10\n",
      " - Found 10650 anagrams of length 11\n",
      " - Found 7194 anagrams of length 12\n",
      " - Found 4434 anagrams of length 13\n",
      " - Found 2459 anagrams of length 14\n",
      " - Found 1384 anagrams of length 15\n",
      " - Found 667 anagrams of length 16\n",
      " - Found 339 anagrams of length 17\n",
      " - Found 128 anagrams of length 18\n",
      " - Found 62 anagrams of length 19\n",
      " - Found 20 anagrams of length 20\n",
      " - Found 9 anagrams of length 21\n",
      " - Found 8 anagrams of length 22\n",
      " - Found 2 anagrams of length 23\n",
      " - Found 1 anagrams of length 24\n",
      "Constructing Language Model...\n",
      " - No language model provided\n"
     ]
    }
   ],
   "source": [
    "model.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df4932e",
   "metadata": {},
   "source": [
    "### Simple Querying\n",
    "\n",
    "Now the model is loaded we can query it as follows, let's take an existing word that's in the model first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c2873e91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'separate', 'score': 1.0, 'dist_score': 1.0, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separated', 'score': 0.8125, 'dist_score': 0.8125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separates', 'score': 0.8125, 'dist_score': 0.8125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separately', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': \"separate's\", 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separative', 'score': 0.734375, 'dist_score': 0.734375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separator', 'score': 0.71875, 'dist_score': 0.71875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separable', 'score': 0.703125, 'dist_score': 0.703125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'serrate', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'disparate', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'aerate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'prate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'operate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'aspirate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'desperate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'spate', 'score': 0.546875, 'dist_score': 0.546875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'spare', 'score': 0.515625, 'dist_score': 0.515625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'serape', 'score': 0.515625, 'dist_score': 0.515625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'seawater', 'score': 0.515625, 'dist_score': 0.515625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n"
     ]
    }
   ],
   "source": [
    "variants  = model.find_variants(\"separate\", SearchParameters())\n",
    "for variant in variants:\n",
    "    print(variant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677c5812",
   "metadata": {},
   "source": [
    "As expected, the word itself is returned with a perfect score of *1.0*, along with various lower-ranking variants. Each variant is represented as a dictionary with the following keys:\n",
    "\n",
    "* ``text`` - The textual value (str) of the variant as it occurs in the lexicon\n",
    "* ``score`` - The combined score of this variant (float), a weighted combination of `dist_score` and `freq_score`\n",
    "* ``dist_score`` - The distance score (float). A perfect match always has score *1.0*.\n",
    "* ``freq_score`` - The frequency score (float), in case lexicons have frequency information. The most frequent match always has score *1.0*.\n",
    "* ``lexicons`` - The lexicons where the match was found (list). This is useful in case you loaded multiple lexicons and may even serve as a simple form of tagging.\n",
    "\n",
    "And let's now try it with misspelled input that is not in the actual model, even though it's not an exact match, we expect the properly spelled variant to come out on top:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "45cacf9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'separate', 'score': 0.734375, 'dist_score': 0.734375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'operate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'desperate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'temperate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'serrate', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'exasperate', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separated', 'score': 0.609375, 'dist_score': 0.609375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'separates', 'score': 0.609375, 'dist_score': 0.609375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'prate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'desecrate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'segregate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'penetrate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'spate', 'score': 0.546875, 'dist_score': 0.546875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'serape', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'secrete', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'operates', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'deprecate', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n",
      "{'text': 'seepage', 'score': 0.515625, 'dist_score': 0.515625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}\n"
     ]
    }
   ],
   "source": [
    "variants = model.find_variants(\"seperate\", SearchParameters())\n",
    "for variant in variants:\n",
    "    print(variant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c4a734f",
   "metadata": {},
   "source": [
    "The `find_variants` method is used to *query* the model directly. Parameters can be specified as part of ``SearchParameters`` using keyword arguments, the following are supported:\n",
    "\n",
    "* ``max_edit_distance`` - Maximum edit distance (levenshtein-damarau). Insertions, deletions, substitutions and transposition all have the same cost (1). It is recommended to set this value slightly lower than the maximum anagram distance. This may take an absolute integer value, i.e. the difference in characters (regardless of order), a floating point value in the range 0-1 to express a relative is expressed ratio of the total length of the text fragment under consideration, or a tuple of a floating point value and an integer (same interpretation as above) with the integer acting as a limit.\n",
    "* ``max_anagram_distance`` - Maximum anagram distance (e heuristic approximation of edit distance). This may take an absolute integer value, i.e. the difference in characters (regardless of order), a floating point value in the range 0-1 to express a relative is expressed ratio of the total length of the text fragment under consideration, or a tuple of a floating point value and an integer (same interpretation as above) with the integer acting as a limit.\n",
    "* ``score_threshold`` - Require scores to meet this threshold (float), they are pruned otherwise\n",
    "* ``cutoff_threshold`` - Cut-off threshold: if a score in the ranking is a specific factor greater than the best score, the ranking will be cut-off at that point and the score not included. Should be set to a value like 2.\n",
    "* ``freq_weight`` - Weight attributed to the frequency information in frequency reranking, in relation to the similarity (distance) component. 0 = disabled)\n",
    "* ``max_matches`` - Number of matches to return per input (set to 0 for unlimited if you want to exhaustively return every possibility within the specified anagram and edit distance).\n",
    "\n",
    "Example with some constraining parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "741c4218",
   "metadata": {},
   "outputs": [],
   "source": [
    "variants = model.find_variants(\"seperate\", SearchParameters(max_anagram_distance=2, max_edit_distance=2, max_matches=1))\n",
    "for variant in variants:\n",
    "    print(variant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea6efd7",
   "metadata": {},
   "source": [
    "Note that in these examples, the resulting `freq_score` is always *1.0* (the maximum) because there is no frequency information associated with our lexicon. Our aspell lexicon was just a plain list of words. Analiticcl often produces better results if corpus-derived frequency information is available in the lexicon, these can be provided as simple counts (integers) in the second column of the tab separated format. The system will then be more inclined to pick high-frequent words over those with low frequency. The weight of the frequency component can be set in `freq_weight` in `SearchParameters`, a value of 0.5 would mean that the frequency score is deemed half as strong/important as the distance score, a value of 2 would mean if is twice as strong/important, a value of 0 disables it entirely (default). A value of 0.25 might be a reasonable default to set if you have frequency information in your lexicon(s). If you load multiple lexicons with frequency information, all frequencies must be expressed on the same scale (e.g. derived from the same corpus).\n",
    "\n",
    "The weights for the similarity/distance computations are set via keyword arguments on initialisation of the `Weights` class, which was, if you recall, passed once when we instantiated the `VariantModel`. We distinguish the following weights, all are floating point values, each corresponds to a component in the similarity/distance computation:\n",
    "\n",
    "* ``ld`` - Edit/Levenshtein distance. This is usually the main component (with highest weight) in the similarity computation (default: 0.5)\n",
    "* ``lcs`` - Longest Common Substring (default: 0.125)\n",
    "* ``prefix`` - Prefix match. Looks at common prefixes (default: 0.125)\n",
    "* ``suffix`` - Suffix match. Looks at common suffixes (default: 0.125)\n",
    "* ``case`` - Casing. Looks at uppercase/lowercase differences  (default: 0.125)\n",
    "\n",
    "You can set a value to 0.0 to disable a component. The weights are most easily interpretable if their sum is 1.0, but this is not a requirement (the system will do the normalisation for you).\n",
    "The result of computation using of these weights ends up in the results dictionary under the `dist_score` key."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d3985f",
   "metadata": {},
   "source": [
    "## Searching\n",
    "\n",
    "Using the `find_variants() ` method you query analiticcl's variant model with an exact input string and ask it to correct it as a single unit. This\n",
    "effectively implements the *correction* part of a spelling-correction system, but does not really handle the *detection*\n",
    "aspect that automatically determines which part of the input needs correction in the first place.\n",
    "\n",
    "If you want to *detect* and subsequently *correct* possible errors in running text, you need the `find_all_matches()`  method.\n",
    "The input is running text (a string) and analiticcl will return all matches it can find:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "157f4143",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': 'We', 'offset': {'begin': 0, 'end': 2}, 'variants': [{'text': 'we', 'score': 0.875, 'dist_score': 0.875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'Wei', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'Web', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'Wed', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wee', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'E', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'W', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'awe', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'ewe', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'owe', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'web', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wed', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wen', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wet', 'score': 0.5, 'dist_score': 0.5, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n",
      "{'input': 'would', 'offset': {'begin': 3, 'end': 8}, 'variants': [{'text': 'would', 'score': 1.0, 'dist_score': 1.0, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'woulds', 'score': 0.775, 'dist_score': 0.775, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'could', 'score': 0.725, 'dist_score': 0.725, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'mould', 'score': 0.725, 'dist_score': 0.725, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wound', 'score': 0.7000000000000001, 'dist_score': 0.7000000000000001, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wold', 'score': 0.675, 'dist_score': 0.675, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'world', 'score': 0.675, 'dist_score': 0.675, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wouldst', 'score': 0.675, 'dist_score': 0.675, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'Gould', 'score': 0.6, 'dist_score': 0.6, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'old', 'score': 0.5249999999999999, 'dist_score': 0.5249999999999999, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'wolds', 'score': 0.5249999999999999, 'dist_score': 0.5249999999999999, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n",
      "{'input': 'like', 'offset': {'begin': 9, 'end': 13}, 'variants': [{'text': 'like', 'score': 1.0, 'dist_score': 1.0, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'alike', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'liked', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'liken', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'liker', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'likes', 'score': 0.75, 'dist_score': 0.75, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'dike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'hike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'kike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'mike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'pike', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'lie', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n",
      "{'input': 'seperate', 'offset': {'begin': 14, 'end': 22}, 'variants': [{'text': 'separate', 'score': 0.734375, 'dist_score': 0.734375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'operate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'desperate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'temperate', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'serrate', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'exasperate', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separated', 'score': 0.609375, 'dist_score': 0.609375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separates', 'score': 0.609375, 'dist_score': 0.609375, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'prate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'desecrate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'segregate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'penetrate', 'score': 0.5625, 'dist_score': 0.5625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'spate', 'score': 0.546875, 'dist_score': 0.546875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'serape', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'secrete', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'operates', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'deprecate', 'score': 0.53125, 'dist_score': 0.53125, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'seepage', 'score': 0.515625, 'dist_score': 0.515625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n",
      "{'input': 'beds', 'offset': {'begin': 23, 'end': 27}, 'variants': [{'text': 'beds', 'score': 1.0, 'dist_score': 1.0, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': \"bed's\", 'score': 0.71875, 'dist_score': 0.71875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bed', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'eds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'feds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'beads', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'reds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'teds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'weds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'zeds', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bends', 'score': 0.6875, 'dist_score': 0.6875, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bees', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'beys', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'begs', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bods', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bids', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bets', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'buds', 'score': 0.65625, 'dist_score': 0.65625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'embeds', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'bedsit', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n"
     ]
    }
   ],
   "source": [
    "matches = model.find_all_matches(\"We would like seperate beds\", SearchParameters(unicodeoffsets=True))\n",
    "for match in matches:\n",
    "    print(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d6570f",
   "metadata": {},
   "source": [
    "Each match will refer back to the string using *begin* and *end* offsets (the end is non-inclusive), if `unicodeoffsets` is set in `SearchParameters` then these will be unicode codepoints, otherwise they will be UTF-8 bytes (analiticcl's internal representation). For Python, you almost always will want to set `unicodeoffsets=True` as that corresponds with how Python deals with string indexing.\n",
    "\n",
    "Now you might think that processing words in a sentence is equal to simply splitting the words (i.e. tokenisation) and calling `find_variants()` on each word, but analiticcl actually does way more than that:\n",
    "\n",
    "* Entries in lexicons need not be single words, we support higher-order n-grams. You can pass keyword argument `max_ngram` to `SearchParameters` with an integer value indicating the maximum order of n-grams you want to support (1 = unigrams (default), 2 = bigrams, etc).\n",
    "* The system tries to find the optimal path in case of conflicting solution\n",
    "* Analiticcl does invoke `find_variants()` behind the scenes, but it can parallellise calls to leverage multiple processor cores and speeding up the process. Set keyword parameter `single_thread=True` for `SearchParameters` if you don't want this behaviour.\n",
    "* Analiticcl can solve runons and splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "defa8848",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': 'sep arate', 'offset': {'begin': 14, 'end': 23}, 'variants': [{'text': 'separate', 'score': 0.7499999999999999, 'dist_score': 0.7499999999999999, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separated', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separates', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separative', 'score': 0.5694444444444445, 'dist_score': 0.5694444444444445, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': 'separately', 'score': 0.5694444444444444, 'dist_score': 0.5694444444444444, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}, {'text': \"separate's\", 'score': 0.5694444444444444, 'dist_score': 0.5694444444444444, 'freq_score': 1.0, 'lexicons': ['examples/eng.aspell.lexicon']}]}\n"
     ]
    }
   ],
   "source": [
    "matches = model.find_all_matches(\"We would like sep arate beds\", SearchParameters(unicodeoffsets=True))\n",
    "print(matches[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a5a27c",
   "metadata": {},
   "source": [
    "When using ``find_all_matches()``, The following keyword arguments are supported for `SearchParameters` in addition to the ones already mentioned before:\n",
    "\n",
    "* `consolidate_matches` - (boolean) Consolidate matches and extract a single most likely sequence, if set to `False`, all possible matches (including overlapping ones) are returned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "977c8389",
   "metadata": {},
   "source": [
    "### Variant lists\n",
    "\n",
    "Thus-far we have used a simple lexicon with one word per line. Additionally, we know we can also load lexicons with corpus-derived frequency information. Now we want to show that you can also load explicit variant lists. \n",
    "A variant list explicitly relates spelling variants to preferred forms, and in doing so go a step further than a simple lexicon which only\n",
    "specifies the validated or corpus-derived form.\n",
    "\n",
    "A variant list is *directed* and *weighted*, it specifies a normalised/preferred form first, and then specifies variants and variant scores (and optionally, frequencies). Take the following example (all fields are tab separated):\n",
    "\n",
    "```tsv\n",
    "separate\tseperate\t1.0\tseprate\t1.0 \n",
    "```\n",
    "\n",
    "This states that the preferred word *separate* has two variants (misspellings in this case), and both have a score\n",
    "(0-1) that expresses how likely the variant maps to the preferred word. Let's load this into analiticcl using `read_variants`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9a8c61f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing anagram values for all items in the lexicon...\n",
      " - Found 3 instances\n",
      "Adding all instances to the index...\n",
      " - Found 3 anagrams\n",
      "Creating sorted secondary index...\n",
      "Sorting secondary index...\n",
      " - Found 1 anagrams of length 7\n",
      " - Found 2 anagrams of length 8\n",
      "Constructing Language Model...\n",
      " - No language model provided\n"
     ]
    }
   ],
   "source": [
    "with open(\"example.variantlist.tsv\",\"w\",encoding=\"utf-8\") as f:\n",
    "    f.write(\"separate\tseperate\t1.0\tseprate\t1.0\\n\")\n",
    "\n",
    "model2 = VariantModel(alphabet_file, Weights())\n",
    "model2.read_variants(\"example.variantlist.tsv\", transparent=True)\n",
    "model2.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61db201",
   "metadata": {},
   "source": [
    "A variant list can be either transparent or not. If it is transparent it means that all the variants it lists will not be returned (they are transparent), only the preferred form is. Such a list is also sometimes known as an error list.\n",
    "If a list is non-transparent, the variants may be return as valid matches.\n",
    "\n",
    "Now if we query for the misspelling *seperate* we get a *perfect* (1.0) match via the variant list, the `via` key expresses that what it was matched through transparently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5eea1cf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'separate', 'score': 1.0, 'dist_score': 1.0, 'freq_score': 1.0, 'via': 'seperate', 'lexicons': ['example.variantlist.tsv']}\n"
     ]
    }
   ],
   "source": [
    "variants = model2.find_variants(\"seperate\", SearchParameters(max_anagram_distance=2, max_edit_distance=2, max_matches=1))\n",
    "for variant in variants:\n",
    "    print(variant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f796f7",
   "metadata": {},
   "source": [
    "Why would you want to use a variant list? We've seen that even with a simple lexicon we were already able to correctly catch this misspelling, Analiticcl after all is precisely designed to match variants without needing to make them explicit.\n",
    "Still a variant list may help especially to bridge larger edit distances without needing to increase the `max_edit_distance` or `max_anagram_distance` of the algorithm (which comes with a performance penalty). It is especially useful in cases of error correction or normalisation where there is a large difference between the preferred form and the (possibly erroneous) variant. It can also be used to link entirely different orthographic synonyms to a preferred normalised form."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76d18fe",
   "metadata": {},
   "source": [
    "### Background Lexicon\n",
    "\n",
    "We can not understate the importance of the background lexicon to reduce false positives. Analiticcl will eagerly\n",
    "attempt to match your test input to whatever lexicons or variant lists you provide. This demands a certain degree of completeness in your\n",
    "lexicons. If your lexicon contains a relatively rare word like \"boulder\" and not a more common word like \"builder\", then\n",
    "analiticcl will happily suggest all instances of \"builder\" to be \"boulder\". The risk for this increases as the allowed\n",
    "edit distances increase. \n",
    "\n",
    "With `model2` from the previous section we now have a good example to illustrate this. This model contains only one entry from a variant list and has no further background lexicon, so if we query it with for example the word *operate* (fairly close in edit distance to the misspelling *seperate* that is in our variant list, and we get *separate* as result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6911b343",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'separate', 'score': 0.625, 'dist_score': 0.625, 'freq_score': 1.0, 'via': 'seprate', 'lexicons': ['example.variantlist.tsv']}\n"
     ]
    }
   ],
   "source": [
    "variants = model2.find_variants(\"operate\", SearchParameters(max_anagram_distance=2, max_edit_distance=2, max_matches=1))\n",
    "for variant in variants:\n",
    "    print(variant)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd0adfc9",
   "metadata": {},
   "source": [
    "Background lexicons should also contain morphological variants and not just lemmas. Ideally it is derived automatically from a fully spell-checked corpus. Analiticcl **will not** work for you if you just feed it some small lexicons and no complete enough background lexicons, unless you are sure your test texts have a very constrained limited vocabulary.\n",
    "\n",
    "### Confusion lists\n",
    "\n",
    "When analiticcl computes distances, it relies on the *alphabet* you provided. Whatever character in the alphabet is substituted for whatever other in the matching process, it carries the same weight for the distance algorithm. However, this is not the case in practice. We often see that certain characters are more often confused than others. \n",
    "\n",
    "Take a word like *analysis*, we may imagine people misspelling it as *analisys*, *analysys* or *analisis*, considering the *i* and *y* have the same phonetic expression in this context.\n",
    "Analiticcl allows you to express such *confusables* in a *confusable list*.\n",
    "\n",
    "The confusable list is a TSV file (tab separated fields) containing known confusable patterns and weights to assign to\n",
    "these patterns when they are found. The file contains one confusable pattern per line. The patterns are expressed in the\n",
    "edit script language of [sesdiff](https://github.com/proycon/sesdiff). Consider the following example:\n",
    "\n",
    "```tsv\n",
    "-[y]+[i]\t1.1\n",
    "```\n",
    "\n",
    "This pattern expressed a deletion of the letter ``y`` followed by insertion of ``i``, which comes down to substitution\n",
    "of ``y`` for  ``i``. Edits that match against this confusable pattern receive the weight *1.1*, meaning such an edit is\n",
    "given preference over edits with other confusable patterns, which by definition have weight *1.0*. Weights greater than\n",
    "*1.0* are being given preference in the score weighting, weights smaller than ``1.0`` imply a penalty. When multiple\n",
    "confusable patterns match, the products of their weights is taken. The final weight is applied to the whole candidate\n",
    "score, so weights should be values fairly close to ``1.0`` in order not to introduce too large bonuses/penalties.\n",
    "\n",
    "The edit script language from sesdiff also allows for matching on immediate context, consider the following variant of the above\n",
    "which only matches the substitution when it comes between two *s* characters (like in *analysys* -> *analysis*) .\n",
    "\n",
    "```tsv\n",
    "=[s]-[y]+[i]=[s]    1.1\n",
    "```\n",
    "\n",
    "To force matches on the beginning or end, start or end the pattern with respectively a  ``^`` or a ``$``. A further description of the edit script language\n",
    "can be found in the [sesdiff](https://github.com/proycon/sesdiff) documentation.\n",
    "\n",
    "A confusion list is loaded using the `read_confusablelist(filename)` method.\n",
    "\n",
    "## Context information\n",
    "\n",
    "In everything that we've seen thus-far, except for the confusion lists and n-grams, context does not play a role yet. However, context is often one of the most important cues in language.\n",
    "There are two ways to take context into account when searching via `find_all_matches()`: language modelling and/or via context rules.\n",
    "\n",
    "### Language modelling\n",
    "\n",
    "In order to consider context information, analiticcl can construct and apply a simple n-gram language model. The input for this language\n",
    "model is an n-gram frequency list, provided through `read_lm(filename)` .\n",
    "\n",
    "The input file should be a corpus-derived list of unigrams and bigrams, optionally also trigrams (and even all up to quintgrams if\n",
    "needed, higher-order ngrams are not supported though).  This is a TSV file containing the ngram in the first column\n",
    "(space character acts as token separator), and the absolute frequency count in the second column. It is also recommended\n",
    "it contains the special tokens ``<bos>`` (begin of sentence) and ``<eos>`` end of sentence. The items in this list are\n",
    "**NOT** used for variant matching, use ``read_lexicon()`` in addition if you want to also match against\n",
    "these items. It is fine to have an entry in both the language model and lexicon, it will be stored only once\n",
    "internally to save memory.\n",
    "\n",
    "When language modelling is used, the following weights for `SearchParameters` become relevant and determine the balance between the language model and the variant model:\n",
    "\n",
    "* ``lm_weight`` - The weight of the language model (float, default 1.0)\n",
    "* ``variantmodel_weight`` - The weight of the variant model (float, default 3.0)\n",
    "\n",
    "Note that even in the variant model, you *may* have a frequency component, whilst frequency information is also expressed in the language model. Take this into account when assigning weights if you don't want frequency information to take on too strong a role. Assigning weights is not trivial and not an exact science, it's a balancing act and values are best determined experimentally by trying out some and seeing what works best for you. Typically the variant model should carry more weight than the language model, otherwise the language model will simply choose fortuitous words that have little relation with the original input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce05f26d",
   "metadata": {},
   "source": [
    "### Context rules\n",
    "\n",
    "Another way to consider context information is through context rules. The context rules define certain patterns that are\n",
    "to be either favoured or penalized (similar to the confusible lists we saw before). The context rules are expressed in a tab separated file which can be passed to\n",
    "analiticcl using ``read_contextrules(filename)``. The first column contains a sequence separated by semicolons, and the second a\n",
    "score close to 1.0 (lower scores penalize the pattern, higher scores favour it):\n",
    "\n",
    "```tsv\n",
    "hello ; world\t1.1\n",
    "```\n",
    "\n",
    "This means that if the words \"hello world\" appear as a solution a text/sentence, its total context score will be boosted\n",
    "(proportional to the length of the match), effectively preferring this solution over others. This context score is an\n",
    "independent component in the final score function and its weight can be set using ``contextrules_weight`` in ``SearchParameters``. Its value is relative to both `lm_weight` and `variantmodel_weight`.\n",
    "\n",
    "Note that the words also need to be in a lexicon you provide for a rule to work. You can express disjunctions using the\n",
    "pipe character (``|``), as follows:\n",
    "\n",
    "```tsv\n",
    "hello|hi ; world|planet\t1.1\n",
    "```\n",
    "\n",
    "This will match all four possible combinations. Rather than match the text, you can match specific lexicons you loaded\n",
    "using the `@` prefix. This makes sense mainly if you use different lexicons and could be used as a form of elementary tagging:\n",
    "\n",
    "```tsv\n",
    "@greetings.tsv ; world\t1.1\n",
    "```\n",
    "\n",
    "Here too you can create disjunctions using the pipe character:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd1d6bd1",
   "metadata": {},
   "source": [
    "```tsv\n",
    "@greetings.tsv|@curses.tsv ; world\t1.1\n",
    "```\n",
    "\n",
    "If you want to negate a match, just add ``!`` as a prefix. This also works in combination with ``@``, allowing you to match anything *except* the words from a particular the lexicon. If you want to negate an entire disjunction, use parenthesis like ``!(a|b|c|)``.\n",
    "\n",
    "There are two standalone characters you may use in matching:\n",
    "\n",
    "* ``?`` - Matches anything\n",
    "* ``^`` - Matches anything that does not match with *any* lexicon (i.e. out of vocabulary words)\n",
    "\n",
    "Note that in all cases,  you'll still need to explicitly load the lexicons (or variants lists) using ``read_lexicon()`` or  ``read_variants()``,\n",
    "etc...\n",
    "\n",
    "The rules are applied in the exact order you specify them. Note that a certain words in a text may only match against\n",
    "one pattern (the first that is found). When defining context rules, you'll generally want to specify longer rules before\n",
    "shorter ones, as otherwise the longer rules might never be considered. For example, in the following example, the second\n",
    "pattern would never apply because the first one already matches:\n",
    "\n",
    "```tsv\n",
    "hello\t1.1\n",
    "hello ; world\t1.1\n",
    "```\n",
    "\n",
    "### Entity Tagging\n",
    "\n",
    "Analiticcl can be used as a simple entity tagger using its context rules. Make sure you understand the above section before you\n",
    "continue reading.\n",
    "\n",
    "You may pass two additional tab-separated columns to the context rules file, the third column specifies a tag to assign\n",
    "to any matches, and an *optional* fourth column specifies an offset for tagging (more about this later). For example:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11bd1464",
   "metadata": {},
   "source": [
    "```tsv\n",
    "hello ; world\t1.1\tgreeting\n",
    "```\n",
    "\n",
    "Any instances of \"hello world\" will be assigned the tag \"greeting\", more specifically \"hello\" will be assigned the tag\n",
    "\"greeting\" and gets sequence number 0, \"world\" gets the same tag and sequence number 1.\n",
    "\n",
    "If you want to tag only a subset and leave certain left or right context untagged, then you can do so by specifying an\n",
    "offset (in matches aka words, not characters). Such an offset takes the form ``offset:length``. For example:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af9920e",
   "metadata": {},
   "source": [
    "```tsv\n",
    "hello ; world\t1.1\tgreeting\t1:1\n",
    "```\n",
    "\n",
    "In this case only the word \"world\" will get the tag greeting (and sequence number 0).\n",
    "\n",
    "It is also possible to assign multiple (even overlapping) tags with a single context rule. Use a semicolon to separate multiple tags and multiplet tag offsets (must be equal amount). However, it is not possible to apply multiple context rules once one has matched:\n",
    "\n",
    "```tsv\n",
    "@firstname.tsv ; @lastname.tsv\t1.0\tperson;firstname;lastname 0:2;0:1;1:2\n",
    "```\n",
    "\n",
    "This mechanism can also be used to assign tags based on lexicons whilst allowing some form of lexicon weighting, even if\n",
    "no further context is included:\n",
    "\n",
    "```tsv\n",
    "@greetings.tsv\t1.0\tgreeting\n",
    "in|to|from ; @city.tsv\t1.1\tlocation\t1:1\n",
    "@firstname.tsv ; @lastname.tsv\t1.0\tperson\n",
    "```\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}