Tokenization notebook: fix step_size calculation

notebooks/document-chunking/tokenization.ipynb

@@ -209,7 +209,7 @@
     "\n",
     "Currently there is a limitation that [only the first 512 tokens are considered](https://www.elastic.co/guide/en/machine-learning/8.12/ml-nlp-limitations.html#ml-nlp-elser-v1-limit-512). To work around this, we can first split the input text into chunks of 512 tokens and feed the chunks to Elasticsearch separately. Actually, we need to use a limit of 510 to leave space for the two special tokens (`[CLS]` and `[SEP]`) that we saw.\n",
     "\n",
-    "Furthermore, in practice we often see improved performance when using overlapping chunks. With ELSER, we recommend 50% token overlap (i.e. a 255 token stride)."
+    "Furthermore, in practice we often see improved performance when using overlapping chunks. With ELSER, we recommend 50% token overlap."
    ]
   },
   {
@@ -223,12 +223,21 @@
     "\n",
     "\n",
     "def chunk(\n",
-    "    tokens, chunk_size=SEMANTIC_SEARCH_TOKEN_LIMIT, overlap_ratio=ELSER_TOKEN_OVERLAP\n",
+    "    text, chunk_size=SEMANTIC_SEARCH_TOKEN_LIMIT, overlap_ratio=ELSER_TOKEN_OVERLAP\n",
     "):\n",
-    "    step_size = round(chunk_size * overlap_ratio)\n",
+    "    step_size = round(chunk_size * (1 - overlap_ratio))\n",
     "\n",
+    "    tokens = bert_tokenizer.encode(text)\n",
+    "    tokens = tokens[1:-1]  # remove special beginning and end tokens\n",
+    "\n",
+    "    result = []\n",
     "    for i in range(0, len(tokens), step_size):\n",
-    "        yield tokens[i : i + chunk_size]"
+    "        end = i + chunk_size\n",
+    "        chunk = tokens[i:end]\n",
+    "        result.append(bert_tokenizer.decode(chunk))\n",
+    "        if end >= len(tokens):\n",
+    "            break\n",
+    "    return result"
    ]
   },
   {
@@ -276,8 +285,7 @@
        "['old major, the old boar on the manor farm, calls the animals on the farm for a meeting, where he compares the humans to parasites and teaches the animals a revolutionary song,\\'beasts of england \\'. when major dies, two young pigs, snowball and napoleon, assume command and turn his dream into a philosophy. the animals revolt and drive the drunken and irresponsible mr jones from the farm, renaming it \" animal farm \". they adopt seven commandments of animal - ism, the most important of which is, \" all animals are equal \". snowball attempts to teach the animals reading and writing ; food is plentiful, and the farm runs smoothly. the pigs elevate themselves to positions of leadership and set aside special food items, ostensibly for their personal health. napoleon takes the pups from the farm dogs and trains them privately. napoleon and snowball struggle for leadership. when snowball announces his plans to build a windmill, napoleon has his dogs chase snowball away and declares himself leader. napoleon enacts changes to the governance structure of the farm, replacing meetings with a committee of pigs, who will run the farm. using a young pig named squealer as a \" mouthpiece \", napoleon claims credit for the windmill idea. the animals work harder with the promise of easier lives with the windmill. after a violent storm, the animals find the windmill annihilated. napoleon and squealer convince the animals that snowball destroyed it, although the scorn of the neighbouring farmers suggests that its walls were too thin. once snowball becomes a scapegoat, napoleon begins purging the farm with his dogs, killing animals he accuses of consorting with his old rival. he and the pigs abuse their power, imposing more control while reserving privileges for themselves and rewriting history, villainising snowball and glorifying napoleon. squealer justifies every statement napoleon makes, even the pigs\\'alteration of the seven commandments of animalism to benefit themselves.\\'beasts of england\\'is replaced by an anthem glorifying napoleon, who appears to be adopting the lifestyle of a man. the animals remain convinced that they are better off than they were when under mr jones. squealer abuses the animals\\'poor memories and invents numbers to show their improvement. mr frederick, one of the neighbouring farmers, attacks the farm, using blasting powder to blow up the restored windmill. though the animals win the battle, they do so at great',\n",
        " 'windmill idea. the animals work harder with the promise of easier lives with the windmill. after a violent storm, the animals find the windmill annihilated. napoleon and squealer convince the animals that snowball destroyed it, although the scorn of the neighbouring farmers suggests that its walls were too thin. once snowball becomes a scapegoat, napoleon begins purging the farm with his dogs, killing animals he accuses of consorting with his old rival. he and the pigs abuse their power, imposing more control while reserving privileges for themselves and rewriting history, villainising snowball and glorifying napoleon. squealer justifies every statement napoleon makes, even the pigs\\'alteration of the seven commandments of animalism to benefit themselves.\\'beasts of england\\'is replaced by an anthem glorifying napoleon, who appears to be adopting the lifestyle of a man. the animals remain convinced that they are better off than they were when under mr jones. squealer abuses the animals\\'poor memories and invents numbers to show their improvement. mr frederick, one of the neighbouring farmers, attacks the farm, using blasting powder to blow up the restored windmill. though the animals win the battle, they do so at great cost, as many, including boxer the workhorse, are wounded. despite his injuries, boxer continues working harder and harder, until he collapses while working on the windmill. napoleon sends for a van to take boxer to the veterinary surgeon\\'s, explaining that better care can be given there. benjamin, the cynical donkey, who \" could read as well as any pig \", notices that the van belongs to a knacker, and attempts to mount a rescue ; but the animals\\'attempts are futile. squealer reports that the van was purchased by the hospital and the writing from the previous owner had not been repainted. he recounts a tale of boxer\\'s death in the hands of the best medical care. years pass, and the pigs learn to walk upright, carry whips and wear clothes. the seven commandments are reduced to a single phrase : \" all animals are equal, but some animals are more equal than others \". napoleon holds a dinner party for the pigs and the humans of the area, who congratulate napoleon on having the hardest - working but least fed animals in the country. napoleon announces an alliance with the humans, against the labouring classes of both \" worlds \". he abolishes practices and traditions related to the revolution,',\n",
        " 'cost, as many, including boxer the workhorse, are wounded. despite his injuries, boxer continues working harder and harder, until he collapses while working on the windmill. napoleon sends for a van to take boxer to the veterinary surgeon\\'s, explaining that better care can be given there. benjamin, the cynical donkey, who \" could read as well as any pig \", notices that the van belongs to a knacker, and attempts to mount a rescue ; but the animals\\'attempts are futile. squealer reports that the van was purchased by the hospital and the writing from the previous owner had not been repainted. he recounts a tale of boxer\\'s death in the hands of the best medical care. years pass, and the pigs learn to walk upright, carry whips and wear clothes. the seven commandments are reduced to a single phrase : \" all animals are equal, but some animals are more equal than others \". napoleon holds a dinner party for the pigs and the humans of the area, who congratulate napoleon on having the hardest - working but least fed animals in the country. napoleon announces an alliance with the humans, against the labouring classes of both \" worlds \". he abolishes practices and traditions related to the revolution, and changes the name of the farm to \" the manor farm \". the animals, overhearing the conversation, notice that the faces of the pigs have begun changing. during a poker match, an argument breaks out between napoleon and mr pilkington, and the animals realise that the faces of the pigs look like the faces of humans, and no one can tell the difference between them. the pigs snowball, napoleon, and squealer adapt old major\\'s ideas into an actual philosophy, which they formally name animalism. soon after, napoleon and squealer indulge in the vices of humans ( drinking alcohol, sleeping in beds, trading ). squealer is employed to alter the seven commandments to account for this humanisation, an allusion to the soviet government\\'s revising of history in order to exercise control of the people\\'s beliefs about themselves and their society. the original commandments are : # whatever goes upon two legs is an enemy. # whatever goes upon four legs, or has wings, is a friend. # no animal shall wear clothes. # no animal shall sleep in a bed. # no animal shall drink alcohol. # no animal shall kill any other animal. # all animals are equal.',\n",
-       " 'and changes the name of the farm to \" the manor farm \". the animals, overhearing the conversation, notice that the faces of the pigs have begun changing. during a poker match, an argument breaks out between napoleon and mr pilkington, and the animals realise that the faces of the pigs look like the faces of humans, and no one can tell the difference between them. the pigs snowball, napoleon, and squealer adapt old major\\'s ideas into an actual philosophy, which they formally name animalism. soon after, napoleon and squealer indulge in the vices of humans ( drinking alcohol, sleeping in beds, trading ). squealer is employed to alter the seven commandments to account for this humanisation, an allusion to the soviet government\\'s revising of history in order to exercise control of the people\\'s beliefs about themselves and their society. the original commandments are : # whatever goes upon two legs is an enemy. # whatever goes upon four legs, or has wings, is a friend. # no animal shall wear clothes. # no animal shall sleep in a bed. # no animal shall drink alcohol. # no animal shall kill any other animal. # all animals are equal. later, napoleon and his pigs secretly revise some commandments to clear them of accusations of law - breaking ( such as \" no animal shall drink alcohol \" having \" to excess \" appended to it and \" no animal shall sleep in a bed \" with \" with sheets \" added to it ). the changed commandments are as follows, with the changes bolded : * 4 no animal shall sleep in a bed with sheets. * 5 no animal shall drink alcohol to excess. * 6 no animal shall kill any other animal without cause. eventually these are replaced with the maxims, \" all animals are equal, but some animals are more equal than others \", and \" four legs good, two legs better! \" as the pigs become more human. this is an ironic twist to the original purpose of the seven commandments, which were supposed to keep order within animal farm by uniting the animals together against the humans, and prevent animals from following the humans\\'evil habits. through the revision of the commandments, orwell demonstrates how simply political dogma can be turned into malleable propaganda.',\n",
-       " 'later, napoleon and his pigs secretly revise some commandments to clear them of accusations of law - breaking ( such as \" no animal shall drink alcohol \" having \" to excess \" appended to it and \" no animal shall sleep in a bed \" with \" with sheets \" added to it ). the changed commandments are as follows, with the changes bolded : * 4 no animal shall sleep in a bed with sheets. * 5 no animal shall drink alcohol to excess. * 6 no animal shall kill any other animal without cause. eventually these are replaced with the maxims, \" all animals are equal, but some animals are more equal than others \", and \" four legs good, two legs better! \" as the pigs become more human. this is an ironic twist to the original purpose of the seven commandments, which were supposed to keep order within animal farm by uniting the animals together against the humans, and prevent animals from following the humans\\'evil habits. through the revision of the commandments, orwell demonstrates how simply political dogma can be turned into malleable propaganda.']"
+       " 'and changes the name of the farm to \" the manor farm \". the animals, overhearing the conversation, notice that the faces of the pigs have begun changing. during a poker match, an argument breaks out between napoleon and mr pilkington, and the animals realise that the faces of the pigs look like the faces of humans, and no one can tell the difference between them. the pigs snowball, napoleon, and squealer adapt old major\\'s ideas into an actual philosophy, which they formally name animalism. soon after, napoleon and squealer indulge in the vices of humans ( drinking alcohol, sleeping in beds, trading ). squealer is employed to alter the seven commandments to account for this humanisation, an allusion to the soviet government\\'s revising of history in order to exercise control of the people\\'s beliefs about themselves and their society. the original commandments are : # whatever goes upon two legs is an enemy. # whatever goes upon four legs, or has wings, is a friend. # no animal shall wear clothes. # no animal shall sleep in a bed. # no animal shall drink alcohol. # no animal shall kill any other animal. # all animals are equal. later, napoleon and his pigs secretly revise some commandments to clear them of accusations of law - breaking ( such as \" no animal shall drink alcohol \" having \" to excess \" appended to it and \" no animal shall sleep in a bed \" with \" with sheets \" added to it ). the changed commandments are as follows, with the changes bolded : * 4 no animal shall sleep in a bed with sheets. * 5 no animal shall drink alcohol to excess. * 6 no animal shall kill any other animal without cause. eventually these are replaced with the maxims, \" all animals are equal, but some animals are more equal than others \", and \" four legs good, two legs better! \" as the pigs become more human. this is an ironic twist to the original purpose of the seven commandments, which were supposed to keep order within animal farm by uniting the animals together against the humans, and prevent animals from following the humans\\'evil habits. through the revision of the commandments, orwell demonstrates how simply political dogma can be turned into malleable propaganda.']"
       ]
      },
      "execution_count": 9,
@@ -286,11 +294,7 @@
     }
    ],
    "source": [
-    "tokens = bert_tokenizer.encode(long_text)[\n",
-    "    1:-1\n",
-    "]  # exclude special tokens at the beginning and end\n",
-    "chunked = [bert_tokenizer.decode(tokens_chunk) for tokens_chunk in chunk(tokens)]\n",
-    "chunked"
+    "chunk(long_text)"
    ]
   },
   {