16-Summarization.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4cRE8IbIrIV"
      },
      "source": [
        "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets as well as other dependencies. Uncomment the following cell and run it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MOsHUjgdIrIW"
      },
      "outputs": [],
      "source": [
        "#! pip install datasets transformers rouge-score nltk"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFuVl8-MbqRN"
      },
      "source": [
        "If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.\n",
        "\n",
        "To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow.\n",
        "\n",
        "First you have to store your authentication token from the Hugging Face website (sign up [here](https://huggingface.co/join) if you haven't already!) then execute the following cell and input your username and password:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "89ry7VzsbqRO"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import notebook_login\n",
        "\n",
        "notebook_login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDpQk-nrbqRP"
      },
      "source": [
        "Then you need to install Git-LFS. Uncomment the following instructions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aIAi-clibqRP"
      },
      "outputs": [],
      "source": [
        "# !apt install git-lfs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fGIyFA1LbqRP"
      },
      "source": [
        "Make sure your version of Transformers is at least 4.11.0 since the functionality was introduced in that version:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xupKwZZabqRQ"
      },
      "outputs": [],
      "source": [
        "import transformers\n",
        "\n",
        "print(transformers.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFASsisvIrIb"
      },
      "source": [
        "You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEJBSTyZIrIb"
      },
      "source": [
        "# Fine-tuning a model on a summarization task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTCFado4IrIc"
      },
      "source": [
        "In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) model for a summarization task. We will use the [XSum dataset](https://arxiv.org/pdf/1808.08745.pdf) (for extreme summarization) which contains BBC articles accompanied with single-sentence summaries.\n",
        "\n",
        "![Widget inference on a summarization task](https://github.com/huggingface/notebooks/blob/master/examples/images/summarization.png?raw=1)\n",
        "\n",
        "We will see how to easily load the dataset for this task using 🤗 Datasets and how to fine-tune a model on it using the `Trainer` API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7KGYpRVfbqRS"
      },
      "outputs": [],
      "source": [
        "model_checkpoint = \"t5-small\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4RRkXuteIrIh"
      },
      "source": [
        "This notebook is built to run  with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a sequence-to-sequence version in the Transformers library. Here we picked the [`t5-small`](https://huggingface.co/t5-small) checkpoint. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whPRbBNbIrIl"
      },
      "source": [
        "## Loading the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7QYTpxXIrIl"
      },
      "source": [
        "We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the data and get the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the functions `load_dataset` and `load_metric`.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IreSlFmlIrIm",
        "outputId": "ff9ab3a2-5c0d-4002-f8ad-1877bbb546a3"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration default\n",
            "Reusing dataset xsum (/home/sgugger/.cache/huggingface/datasets/xsum/default/1.2.0/f9abaabb5e2b2a1e765c25417264722d31877b34ec34b437c53242f6e5c30d6d)\n"
          ]
        }
      ],
      "source": [
        "from datasets import load_dataset, load_metric\n",
        "\n",
        "raw_datasets = load_dataset(\"xsum\")\n",
        "metric = load_metric(\"rouge\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzfPtOMoIrIu"
      },
      "source": [
        "The `dataset` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GWiVUF0jIrIv",
        "outputId": "35e3ea43-f397-4a54-c90c-f2cf8d36873e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['document', 'summary', 'id'],\n",
              "        num_rows: 204045\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['document', 'summary', 'id'],\n",
              "        num_rows: 11332\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['document', 'summary', 'id'],\n",
              "        num_rows: 11334\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "raw_datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3EtYfeHIrIz"
      },
      "source": [
        "To access an actual element, you need to select a split first, then give an index:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X6HrpprwIrIz",
        "outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'document': 'Recent reports have linked some France-based players with returns to Wales.\\n\"I\\'ve always felt - and this is with my rugby hat on now; this is not region or WRU - I\\'d rather spend that money on keeping players in Wales,\" said Davies.\\nThe WRU provides £2m to the fund and £1.3m comes from the regions.\\nFormer Wales and British and Irish Lions fly-half Davies became WRU chairman on Tuesday 21 October, succeeding deposed David Pickering following governing body elections.\\nHe is now serving a notice period to leave his role as Newport Gwent Dragons chief executive after being voted on to the WRU board in September.\\nDavies was among the leading figures among Dragons, Ospreys, Scarlets and Cardiff Blues officials who were embroiled in a protracted dispute with the WRU that ended in a £60m deal in August this year.\\nIn the wake of that deal being done, Davies said the £3.3m should be spent on ensuring current Wales-based stars remain there.\\nIn recent weeks, Racing Metro flanker Dan Lydiate was linked with returning to Wales.\\nLikewise the Paris club\\'s scrum-half Mike Phillips and centre Jamie Roberts were also touted for possible returns.\\nWales coach Warren Gatland has said: \"We haven\\'t instigated contact with the players.\\n\"But we are aware that one or two of them are keen to return to Wales sooner rather than later.\"\\nSpeaking to Scrum V on BBC Radio Wales, Davies re-iterated his stance, saying keeping players such as Scarlets full-back Liam Williams and Ospreys flanker Justin Tipuric in Wales should take precedence.\\n\"It\\'s obviously a limited amount of money [available]. The union are contributing 60% of that contract and the regions are putting £1.3m in.\\n\"So it\\'s a total pot of just over £3m and if you look at the sorts of salaries that the... guys... have been tempted to go overseas for [are] significant amounts of money.\\n\"So if we were to bring the players back, we\\'d probably get five or six players.\\n\"And I\\'ve always felt - and this is with my rugby hat on now; this is not region or WRU - I\\'d rather spend that money on keeping players in Wales.\\n\"There are players coming out of contract, perhaps in the next year or so… you\\'re looking at your Liam Williams\\' of the world; Justin Tipuric for example - we need to keep these guys in Wales.\\n\"We actually want them there. They are the ones who are going to impress the young kids, for example.\\n\"They are the sort of heroes that our young kids want to emulate.\\n\"So I would start off [by saying] with the limited pot of money, we have to retain players in Wales.\\n\"Now, if that can be done and there\\'s some spare monies available at the end, yes, let\\'s look to bring players back.\\n\"But it\\'s a cruel world, isn\\'t it?\\n\"It\\'s fine to take the buck and go, but great if you can get them back as well, provided there\\'s enough money.\"\\nBritish and Irish Lions centre Roberts has insisted he will see out his Racing Metro contract.\\nHe and Phillips also earlier dismissed the idea of leaving Paris.\\nRoberts also admitted being hurt by comments in French Newspaper L\\'Equipe attributed to Racing Coach Laurent Labit questioning their effectiveness.\\nCentre Roberts and flanker Lydiate joined Racing ahead of the 2013-14 season while scrum-half Phillips moved there in December 2013 after being dismissed for disciplinary reasons by former club Bayonne.',\n",
              " 'id': '29750031',\n",
              " 'summary': 'New Welsh Rugby Union chairman Gareth Davies believes a joint £3.3m WRU-regions fund should be used to retain home-based talent such as Liam Williams, not bring back exiled stars.'}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "raw_datasets[\"train\"][0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHUmphG3IrI3"
      },
      "source": [
        "To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3j8APAoIrI3"
      },
      "outputs": [],
      "source": [
        "import datasets\n",
        "import random\n",
        "import pandas as pd\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "def show_random_elements(dataset, num_examples=5):\n",
        "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
        "    picks = []\n",
        "    for _ in range(num_examples):\n",
        "        pick = random.randint(0, len(dataset)-1)\n",
        "        while pick in picks:\n",
        "            pick = random.randint(0, len(dataset)-1)\n",
        "        picks.append(pick)\n",
        "    \n",
        "    df = pd.DataFrame(dataset[picks])\n",
        "    for column, typ in dataset.features.items():\n",
        "        if isinstance(typ, datasets.ClassLabel):\n",
        "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
        "    display(HTML(df.to_html()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SZy5tRB_IrI7",
        "outputId": "ba8f2124-e485-488f-8c0c-254f34f24f13"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document</th>\n",
              "      <th>id</th>\n",
              "      <th>summary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Cheryl Boone Isaacs said that the relationship with the accountancy firm PriceWaterhouseCoopers (PWC) was also under review.\\nBrian Cullinan and Martha Ruiz were responsible for Sunday's mishap.\\nLa La Land was mistakenly announced as the winner of the best picture award.\\nThe team behind the film were in the middle of their speeches before it was revealed the accolade should have gone to Moonlight.\\nIt has been described as the biggest mistake in 89 years of Academy Awards history.\\nHow did the Oscars mistake happen?\\nNine epic awards fails\\nMr Cullinan mistakenly handed the wrong envelope to the two presenters.\\nHe gave Warren Beatty and Faye Dunaway the back-up envelope for best actress in a leading role - rather than the envelope which contained the name of the winner for the best film.\\nPriceWaterhouseCoopers, which counts the votes and organises the envelopes, has apologised for the mix-up.\\nMr Cullinan tweeted a picture of best actress winner Emma Stone minutes before handing the presenters the wrong envelope, and Ms Boone Isaacs blamed \"distraction\" for the error.</td>\n",
              "      <td>39136183</td>\n",
              "      <td>The two accountants responsible for muddling up the main award envelopes at Sunday's Oscars ceremony will not be employed to do the job again, the academy president has announced.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Eddie Bilbey, from Ripley, Derbyshire, died shortly after the bout in South Normanton on 24 March.\\nThe teenager was a Nottingham Forest supporter and the event will reflect this, with ex-players due to attend.\\nOrganiser Rhyce Hallsworth said the Eddie's family had been \"overwhelmed\" by the love and support shown following his death.\\nEddie was competing in an East Midlands youth welterweight contest. He collapsed and later died in hospital.\\nMike Shinfield, the boy's step-grandfather, who owns Somercotes Elite Boxing Academy where Eddie was a member, told the BBC that he was fit and healthy and there were no concerns relating to his health.\\nA post-mortem examination into how the teenager died has taken place, but the cause of death will not be made public until May.\\nMr Hallsworth, who coaches one of Eddie's brothers at football, said: \"The [Bilbey] family have been going through a torrid time and it was a case of what can we do to help?\\n\"They are a lovely family and they help a lot of other people so we needed to do something for them.\\n\"Eddie was a fine young man, dedicated to his sport, always happy to chat and a genuine credit to his family.\"\\nHe said money raised from the event will go towards charity Bloomin Dementia and the Eddie Bilbey Foundation, which has been set up in the teenager's memory.\\nNottingham Forest fans started a round of applause for Eddie during the 17th minute of the match against Huddersfield on 8 April.\\nMr Hallsworth said Reds fans have been \"absolutely fantastic\".\\nThe ticketed event is due to be held at Saltbox, in Nottingham, on Sunday, ahead of Forest's match against Ipswich Town.</td>\n",
              "      <td>39704400</td>\n",
              "      <td>A fundraising event is to be held in memory of a 17-year-old boy who died after an amateur boxing match.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Iran spoke of \"serious issues\" that must be resolved, while one Western diplomat cited \"considerable gaps\".\\nRussian Foreign Minister Sergei Lavrov has arrived to join the talks, and reports say the discussions could extend into the weekend.\\nThe deal could see Iran curb uranium enrichment for some sanctions relief.\\nHowever, US politicians have indicated they will push forward with a bill proposing more sanctions against Iran next month if the talks fail.\\nUS Senate Majority Leader Harry Reid said he would support \"broadening the scope\" of current oil and trade sanctions.\\nP5+1 wants Iran to:\\nIran wants the P5+1 to:\\nQ&amp;A: Iran nuclear crisis\\nIran's nuclear sites\\nAnalysis: Can gaps be bridged?\\nBoth Republican and Democrat congressmen say the threat of sanctions will bolster the negotiating position of the world powers.\\nPresident Barack Obama had earlier urged Congress not to promote the bill while talks were going on.\\nIran insists its nuclear programme is for peaceful purposes, but some world powers suspect it is seeking a nuclear weapons capability.\\nTalks extended?\\nEU foreign policy chief Baroness Catherine Ashton is leading the talks for the world powers.\\nShe has begun a meeting with Iranian Foreign Minister Mohammed Javad Zarif.\\nThe Geneva talks, which are currently scheduled to conclude on Friday, involve Iran and representatives of the so-called P5+1 - UN Security Council permanent members US, UK, France, China and Russia, plus Germany.\\nIranian Deputy Foreign Minister Abbas Araqchi said on Friday that negotiations were \"positive\", but said meetings were likely to continue into Saturday.\\nUS Secretary of State John Kerry may join later.\\nAfter Thursday's talks, Mr Araqchi was quoted by the Mehr news agency as saying \"serious issues remain a source of difference\".\\nOne senior Western diplomat told Reuters: \"Considerable gaps remain, and we have to narrow the gaps. Some issues really need to be clarified.\\n\"I sensed a real commitment... from both sides. Will it happen? We will see. But, as always, the devil is in the details.\"\\nOne US source said simply: \"It is very hard.\"\\nFrench Foreign Minister Laurent Fabius told France 2 television: \"This deal will only be possible if it has a firm base.\"\\nHowever, one diplomat told Reuters there was still a \"very high probability\" that foreign ministers would join the talks at some point.\\nThe Geneva meeting follows a previous round of talks earlier this month.\\nThe US has said any interim agreement would see the bulk of international and US sanctions targeting Iran's nuclear programme remain in place.\\nMr Obama said sanctions relief would be worth between $6bn and $7bn.\\nThe essence of the deal would involve Iran making no more advances in its nuclear programme and agreeing to \"more vigorous inspections\", he said.\\nAnalysts say a major sticking point is Iran's insistence on its right to enrich uranium - a process that yields material used to manufacture fuel for power stations, but can also be used for weapons.\\nWestern diplomats are also concerned about a reactor Iran is building at Arak, which disrupted the first round of talks.\\nBefore the talks opened, Iran's Supreme Leader Ayatollah Khamenei said Iran would not step back \"one iota\" from its nuclear rights.\\nHe also referred to Israel as a \"rabid dog\".\\nIsrael has vehemently opposed the proposed deal and says it will not be obliged to honour it.\\nPrime Minister Benjamin Netanyahu, who is in Moscow, said the ayatollah's speech represented the \"real Iran\".\\n\"We are not confused. They must not have nuclear weapons. And I promise you that they will not have nuclear weapons,\" he said.</td>\n",
              "      <td>25041814</td>\n",
              "      <td>World powers are struggling to reach an interim deal with Iran to limit its nuclear programme, as talks in Geneva move into a third day.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A 17-year-old boy has been arrested. A 14-year-old arrested earlier was released without charge.\\nA spokesperson for the fire service said it was alerted to the blaze at Dawn Paper's premises on Donore Road at around 22:00 local time on Saturday.\\nThe fire service said it expected to have crews at the site for the rest of the day.\\nThe roof of the building has collapsed and what remains of it is expected to smoulder for a number of days.\\nAt one point during the night 12 units of the fire brigade tackled the blaze.\\nThere are no reports of any casualties.\\nDawn Paper was established in 1988 and the company manufactures industrial cleaning paper and domestic tissue paper products for the Irish market.</td>\n",
              "      <td>34238550</td>\n",
              "      <td>A paper factory in Drogheda, County Louth, has been destroyed in a suspected arson attack.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Real name Dennis Princewell Stehr, Mr Probz has sold 127,000 copies of the song, which is his debut UK release.\\nHe is the latest Dutch artist to top the UK singles chart following Bingo Players, Martin Garrix and Nicky Romero, who all scored hits last year.\\nLast week's number one, Canadian singer Kiesza's deep house anthem Hideaway, has dropped to number two.\\nLondon production duo Sigma are at number three with Nobody To Love, a dance reworking of Kanye West's Bound 2.\\nSee the UK Top 40 singles chart\\nSee the UK Top 40 albums chart\\nBBC Radio 1's Official Chart Show\\nUS soul singer John Legend is at four with All Of Me and Cash Cash complete the top five with Take Me Home.\\nNew entries include DJ Cassidy at number six with Calling All Hearts, featuring Robin Thicke and Jessie J, and girl group Neon Jungle at seven with Welcome To The Jungle.\\nOn the albums chart, Scottish singer-songwriter Paolo Nutini has kept the top spot for a second week with his third album Caustic Love.\\nJohn Legend's Love In The Future has climbed to number two, boy band The Vamps' Meet The Vamps is in third place and Paloma Faith is at number four with her third album A Perfect Contradiction.\\nAustralian rapper Iggy Azalea has scored the week's highest new entry with The New Classic at number five.</td>\n",
              "      <td>27182326</td>\n",
              "      <td>Dutch rapper-turned-singer Mr Probz has gone straight to number one on the UK singles chart with his tune Waves.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "show_random_elements(raw_datasets[\"train\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lnjDIuQ3IrI-"
      },
      "source": [
        "The metric is an instance of [`datasets.Metric`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Metric):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5o4rUteaIrI_",
        "outputId": "18038ef5-554c-45c5-e00a-133b02ec10f1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Metric(name: \"rouge\", features: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')}, usage: \"\"\"\n",
              "Calculates average rouge scores for a list of hypotheses and references\n",
              "Args:\n",
              "    predictions: list of predictions to score. Each predictions\n",
              "        should be a string with tokens separated by spaces.\n",
              "    references: list of reference for each prediction. Each\n",
              "        reference should be a string with tokens separated by spaces.\n",
              "    rouge_types: A list of rouge types to calculate.\n",
              "        Valid names:\n",
              "        `\"rouge{n}\"` (e.g. `\"rouge1\"`, `\"rouge2\"`) where: {n} is the n-gram based scoring,\n",
              "        `\"rougeL\"`: Longest common subsequence based scoring.\n",
              "        `\"rougeLSum\"`: rougeLsum splits text using `\"\n",
              "\"`.\n",
              "        See details in https://github.com/huggingface/datasets/issues/617\n",
              "    use_stemmer: Bool indicating whether Porter stemmer should be used to strip word suffixes.\n",
              "    use_agregator: Return aggregates if this is set to True\n",
              "Returns:\n",
              "    rouge1: rouge_1 (precision, recall, f1),\n",
              "    rouge2: rouge_2 (precision, recall, f1),\n",
              "    rougeL: rouge_l (precision, recall, f1),\n",
              "    rougeLsum: rouge_lsum (precision, recall, f1)\n",
              "Examples:\n",
              "\n",
              "    >>> rouge = datasets.load_metric('rouge')\n",
              "    >>> predictions = [\"hello there\", \"general kenobi\"]\n",
              "    >>> references = [\"hello there\", \"general kenobi\"]\n",
              "    >>> results = rouge.compute(predictions=predictions, references=references)\n",
              "    >>> print(list(results.keys()))\n",
              "    ['rouge1', 'rouge2', 'rougeL', 'rougeLsum']\n",
              "    >>> print(results[\"rouge1\"])\n",
              "    AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))\n",
              "    >>> print(results[\"rouge1\"].mid.fmeasure)\n",
              "    1.0\n",
              "\"\"\", stored examples: 0)"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "metric"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAWdqcUBIrJC"
      },
      "source": [
        "You can call its `compute` method with your predictions and labels, which need to be list of decoded strings:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6XN1Rq0aIrJC",
        "outputId": "a4405435-a8a9-41ff-9f79-a13077b587c7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'rouge1': AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0)),\n",
              " 'rouge2': AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0)),\n",
              " 'rougeL': AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0)),\n",
              " 'rougeLsum': AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "fake_preds = [\"hello there\", \"general kenobi\"]\n",
        "fake_labels = [\"hello there\", \"general kenobi\"]\n",
        "metric.compute(predictions=fake_preds, references=fake_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n9qywopnIrJH"
      },
      "source": [
        "## Preprocessing the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVx71GdAIrJH"
      },
      "source": [
        "Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that the model requires.\n",
        "\n",
        "To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
        "\n",
        "- we get a tokenizer that corresponds to the model architecture we want to use,\n",
        "- we download the vocabulary used when pretraining this specific checkpoint.\n",
        "\n",
        "That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eXNLu_-nIrJI"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "    \n",
        "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vl6IidfdIrJK"
      },
      "source": [
        "By default, the call above will use one of the fast tokenizers (backed by Rust) from the 🤗 Tokenizers library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rowT4iCLIrJK"
      },
      "source": [
        "You can directly call this tokenizer on one sentence or a pair of sentences:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a5hBlsrHIrJL",
        "outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [8774, 6, 48, 80, 7142, 55, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer(\"Hello, this one sentence!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qo_0B1M2IrJM"
      },
      "source": [
        "Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in [this tutorial](https://huggingface.co/transformers/preprocessing.html) if you're interested.\n",
        "\n",
        "Instead of one sentence, we can pass along a list of sentences:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4TWPxtZsbqRX",
        "outputId": "cb884849-129e-4e07-b011-ff7adff00e77"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [[8774, 6, 48, 80, 7142, 55, 1], [100, 19, 430, 7142, 5, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer([\"Hello, this one sentence!\", \"This is another sentence.\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sJBqA6DYbqRY"
      },
      "source": [
        "To prepare the targets for our model, we need to tokenize them inside the `as_target_tokenizer` context manager. This will make sure the tokenizer uses the special tokens corresponding to the targets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZqUDRUNjbqRY",
        "outputId": "46cf4107-91f7-4612-8eac-22e642660e7b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'input_ids': [[8774, 6, 48, 80, 7142, 55, 1], [100, 19, 430, 7142, 5, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]}\n"
          ]
        }
      ],
      "source": [
        "with tokenizer.as_target_tokenizer():\n",
        "    print(tokenizer([\"Hello, this one sentence!\", \"This is another sentence.\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2C0hcmp9IrJQ"
      },
      "source": [
        "If you are using one of the five T5 checkpoints we have to prefix the inputs with \"summarize:\" (the model can also translate and it needs the prefix to know which task it has to perform)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Vq0csOybqRY"
      },
      "outputs": [],
      "source": [
        "if model_checkpoint in [\"t5-small\", \"t5-base\", \"t5-larg\", \"t5-3b\", \"t5-11b\"]:\n",
        "    prefix = \"summarize: \"\n",
        "else:\n",
        "    prefix = \"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ali0BX4ibqRZ"
      },
      "source": [
        "We can then write the function that will preprocess our samples. We just feed them to the `tokenizer` with the argument `truncation=True`. This will ensure that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model. The padding will be dealt with later on (in a data collator) so we pad examples to the longest length in the batch and not the whole dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vc0BSBLIIrJQ"
      },
      "outputs": [],
      "source": [
        "max_input_length = 1024\n",
        "max_target_length = 128\n",
        "\n",
        "def preprocess_function(examples):\n",
        "    inputs = [prefix + doc for doc in examples[\"document\"]]\n",
        "    model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)\n",
        "\n",
        "    # Setup the tokenizer for targets\n",
        "    with tokenizer.as_target_tokenizer():\n",
        "        labels = tokenizer(examples[\"summary\"], max_length=max_target_length, truncation=True)\n",
        "\n",
        "    model_inputs[\"labels\"] = labels[\"input_ids\"]\n",
        "    return model_inputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lm8ozrJIrJR"
      },
      "source": [
        "This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-b70jh26IrJS",
        "outputId": "acd3a42d-985b-44ee-9daa-af5d944ce1d9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [[21603, 10, 17716, 2279, 43, 5229, 128, 1410, 18, 390, 1508, 28, 5146, 12, 10256, 5, 96, 196, 31, 162, 373, 1800, 3, 18, 11, 48, 19, 28, 82, 22209, 3, 547, 30, 230, 117, 48, 19, 59, 1719, 42, 549, 8503, 3, 18, 27, 31, 26, 1066, 1492, 24, 540, 30, 2627, 1508, 16, 10256, 976, 243, 28571, 5, 37, 549, 8503, 795, 17586, 51, 12, 8, 3069, 11, 3996, 13606, 51, 639, 45, 8, 6266, 5, 18263, 10256, 11, 2390, 11, 7262, 10371, 7, 3971, 18, 17114, 28571, 1632, 549, 8503, 13404, 30, 2818, 1401, 1797, 6, 7229, 53, 20, 12151, 1955, 8356, 49, 53, 826, 3, 19585, 643, 9768, 5, 216, 19, 230, 3122, 3, 9, 2103, 1059, 12, 1175, 112, 1075, 38, 24260, 350, 16103, 10282, 7, 5752, 4297, 227, 271, 3, 11060, 30, 12, 8, 549, 8503, 1476, 16, 1600, 5, 28571, 47, 859, 8, 1374, 5638, 859, 10282, 7, 6, 411, 7, 2026, 63, 7, 6, 14586, 7677, 11, 26911, 2419, 7, 4298, 113, 130, 10960, 52, 26786, 16, 3, 9, 813, 11674, 11044, 28, 8, 549, 8503, 24, 3492, 16, 3, 9, 3996, 3328, 51, 1154, 16, 1660, 48, 215, 5, 86, 8, 7178, 13, 24, 1154, 271, 612, 6, 28571, 243, 8, 3996, 19660, 51, 225, 36, 1869, 30, 3, 5833, 750, 10256, 18, 390, 4811, 2367, 132, 5, 86, 1100, 1274, 6, 16046, 10730, 24397, 49, 2744, 31914, 17, 15, 47, 5229, 28, 7646, 12, 10256, 5, 3, 21322, 8, 1919, 1886, 31, 7, 14667, 440, 18, 17114, 4794, 16202, 7, 11, 2050, 17845, 2715, 7, 130, 92, 2633, 26, 21, 487, 5146, 5, 10256, 3763, 16700, 2776, 17, 40, 232, 65, 243, 10, 96, 1326, 43, 29, 31, 17, 16, 7, 2880, 920, 574, 28, 8, 1508, 5, 96, 11836, 62, 33, 2718, 24, 80, 42, 192, 13, 135, 33, 9805, 12, 1205, 12, 10256, 14159, 1066, 145, 865, 535, 14734, 12, 4712, 2781, 584, 30, 9938, 5061, 10256, 6, 28571, 3, 60, 18, 155, 15, 4094, 112, 3, 8389, 6, 2145, 2627, 1508, 224, 38, 14586, 7677, 423, 18, 1549, 1414, 265, 6060, 11, 411, 7, 2026, 63, 7, 24397, 49, 12446, 2262, 3791, 447, 16, 10256, 225, 240, 20799, 1433, 5, 96, 196, 17, 31, 7, 6865, 3, 9, 1643, 866, 13, 540, 784, 28843, 4275, 37, 7021, 33, 12932, 15436, 13, 24, 1696, 11, 8, 6266, 33, 3, 3131, 3996, 13606, 51, 16, 5, 96, 5231, 34, 31, 7, 3, 9, 792, 815, 13, 131, 147, 23395, 51, 11, 3, 99, 25, 320, 44, 8, 10549, 13, 21331, 24, 8, 233, 3413, 233, 43, 118, 3, 22765, 12, 281, 10055, 21, 784, 355, 908, 1516, 6201, 13, 540, 5, 96, 5231, 3, 99, 62, 130, 12, 830, 8, 1508, 223, 6, 62, 31, 26, 1077, 129, 874, 42, 1296, 1508, 5, 96, 7175, 27, 31, 162, 373, 1800, 3, 18, 11, 48, 19, 28, 82, 22209, 3, 547, 30, 230, 117, 48, 19, 59, 1719, 42, 549, 8503, 3, 18, 27, 31, 26, 1066, 1492, 24, 540, 30, 2627, 1508, 16, 10256, 5, 96, 7238, 33, 1508, 1107, 91, 13, 1696, 6, 2361, 16, 8, 416, 215, 42, 78, 233, 25, 31, 60, 479, 44, 39, 1414, 265, 6060, 31, 13, 8, 296, 117, 12446, 2262, 3791, 447, 21, 677, 3, 18, 62, 174, 12, 453, 175, 3413, 16, 10256, 5, 96, 1326, 700, 241, 135, 132, 5, 328, 33, 8, 2102, 113, 33, 352, 12, 18514, 8, 1021, 1082, 6, 21, 677, 5, 96, 10273, 33, 8, 1843, 13, 17736, 24, 69, 1021, 1082, 241, 12, 29953, 5, 96, 5231, 27, 133, 456, 326, 784, 969, 2145, 908, 28, 8, 1643, 815, 13, 540, 6, 62, 43, 12, 7365, 1508, 16, 10256, 5, 96, 17527, 6, 3, 99, 24, 54, 36, 612, 11, 132, 31, 7, 128, 8179, 3, 26413, 347, 44, 8, 414, 6, 4273, 6, 752, 31, 7, 320, 12, 830, 1508, 223, 5, 96, 11836, 34, 31, 7, 3, 9, 23958, 296, 6, 19, 29, 31, 17, 34, 58, 96, 196, 17, 31, 7, 1399, 12, 240, 8, 3, 13863, 11, 281, 6, 68, 248, 3, 99, 25, 54, 129, 135, 223, 38, 168, 6, 937, 132, 31, 7, 631, 540, 535, 2390, 11, 7262, 10371, 7, 2050, 2715, 7, 65, 16, 15777, 3, 88, 56, 217, 91, 112, 16046, 10730, 1696, 5, 216, 11, 16202, 7, 92, 2283, 19664, 8, 800, 13, 3140, 1919, 5, 2715, 7, 92, 10246, 271, 4781, 57, 2622, 16, 2379, 29494, 301, 31, 427, 23067, 15, 3, 20923, 12, 16046, 9493, 9906, 17, 325, 2360, 822, 53, 70, 9570, 5, 2969, 2715, 7, 11, 24397, 49, 31914, 17, 15, 3311, 16046, 2177, 13, 8, 2038, 11590, 774, 298, 14667, 440, 18, 17114, 16202, 7, 2301, 132, 16, 1882, 2038, 227, 271, 19664, 21, 3, 15471, 2081, 57, 1798, 1886, 2474, 5993, 5, 1], [21603, 10, 6788, 19758, 7, 2273, 130, 718, 91, 12, 1154, 28, 3, 9, 6220, 2642, 44, 8, 6036, 30, 8, 368, 3540, 986, 7, 2409, 30, 1701, 706, 5, 2409, 7, 130, 16645, 326, 11, 2117, 12355, 1054, 38, 3, 9, 6478, 16813, 47, 4006, 91, 5, 37, 12787, 6, 261, 57, 1932, 27874, 5220, 31195, 3230, 6, 43, 118, 7774, 3, 9, 381, 13, 648, 5, 1377, 1310, 6, 17602, 6417, 6032, 130, 4006, 91, 30, 8, 6036, 30, 12096, 8348, 16, 1186, 11, 932, 5, 37, 6032, 1553, 826, 3, 9, 27874, 896, 2063, 2902, 16, 1882, 1673, 18395, 53, 8, 7070, 13, 8, 7021, 5692, 44, 8, 896, 2501, 5, 1193, 1778, 29, 53, 8, 1251, 3534, 9, 226, 6, 11529, 283, 4569, 4409, 5225, 8692, 243, 10, 96, 196, 17, 19, 3, 9, 2261, 5415, 21, 8, 415, 616, 6, 34, 4110, 2261, 17879, 6, 34, 10762, 151, 31, 7, 1342, 44, 1020, 6, 34, 54, 1709, 3583, 364, 7232, 8, 616, 5, 96, 19494, 62, 174, 151, 28, 251, 12, 698, 24, 28, 8, 2095, 16, 455, 21, 135, 12, 103, 70, 613, 11, 830, 175, 151, 12, 4831, 535, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'labels': [[368, 22982, 24895, 3545, 13404, 3121, 15, 189, 28571, 7228, 3, 9, 4494, 3996, 19660, 51, 549, 8503, 18, 18145, 7, 3069, 225, 36, 261, 12, 7365, 234, 18, 390, 3683, 224, 38, 1414, 265, 6060, 6, 59, 830, 223, 1215, 699, 26, 4811, 5, 1], [71, 21641, 2642, 646, 1067, 46, 11529, 3450, 828, 16, 5727, 27874, 65, 118, 10126, 3, 9, 3534, 9, 226, 5, 1]]}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "preprocess_function(raw_datasets['train'][:2])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zS-6iXTkIrJT"
      },
      "source": [
        "To apply this function on all the pairs of sentences in our dataset, we just use the `map` method of our `dataset` object we created earlier. This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DDtsaJeVIrJT",
        "outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/xsum/default/1.2.0/f9abaabb5e2b2a1e765c25417264722d31877b34ec34b437c53242f6e5c30d6d/cache-798454ba84bcf26e.arrow\n",
            "Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/xsum/default/1.2.0/f9abaabb5e2b2a1e765c25417264722d31877b34ec34b437c53242f6e5c30d6d/cache-7bca00a09a5288ab.arrow\n",
            "Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/xsum/default/1.2.0/f9abaabb5e2b2a1e765c25417264722d31877b34ec34b437c53242f6e5c30d6d/cache-288693030a39fd3d.arrow\n"
          ]
        }
      ],
      "source": [
        "tokenized_datasets = raw_datasets.map(preprocess_function, batched=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voWiw8C7IrJV"
      },
      "source": [
        "Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
        "\n",
        "Note that we passed `batched=True` to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "545PP3o8IrJV"
      },
      "source": [
        "## Fine-tuning the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBiW8UpKIrJW"
      },
      "source": [
        "Now that our data is ready, we can download the pretrained model and fine-tune it. Since our task is of the sequence-to-sequence kind, we use the `AutoModelForSeq2SeqLM` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TlqNaB8jIrJW"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer\n",
        "\n",
        "model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CczA5lJlIrJX"
      },
      "source": [
        "Note that  we don't get a warning like in our classification example. This means we used all the weights of the pretrained model and there is no randomly initialized head in this case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_N8urzhyIrJY"
      },
      "source": [
        "To instantiate a `Seq2SeqTrainer`, we will need to define three more things. The most important is the [`Seq2SeqTrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Seq2SeqTrainingArguments), which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GNRBe6yobqRa"
      },
      "outputs": [],
      "source": [
        "batch_size = 16\n",
        "model_name = model_checkpoint.split(\"/\")[-1]\n",
        "args = Seq2SeqTrainingArguments(\n",
        "    f\"{model_name}-finetuned-xsum\",\n",
        "    evaluation_strategy = \"epoch\",\n",
        "    learning_rate=2e-5,\n",
        "    per_device_train_batch_size=batch_size,\n",
        "    per_device_eval_batch_size=batch_size,\n",
        "    weight_decay=0.01,\n",
        "    save_total_limit=3,\n",
        "    num_train_epochs=1,\n",
        "    predict_with_generate=True,\n",
        "    fp16=True,\n",
        "    push_to_hub=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "km3pGVdTIrJc"
      },
      "source": [
        "Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the `batch_size` defined at the top of the cell and customize the weight decay. Since the `Seq2SeqTrainer` will save the model regularly and our dataset is quite large, we tell it to make three saves maximum. Lastly, we use the `predict_with_generate` option (to properly generate summaries) and activate mixed precision training (to go a bit faster).\n",
        "\n",
        "The last argument to setup everything so we can push the model to the [Hub](https://huggingface.co/models) regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/t5-finetuned-xsum\"` or `\"huggingface/t5-finetuned-xsum\"`).\n",
        "\n",
        "Then, we need a special kind of data collator, which will not only pad the inputs to the maximum length in the batch, but also the labels:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1MHrdpifbqRb"
      },
      "outputs": [],
      "source": [
        "data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7sZOdRlRIrJd"
      },
      "source": [
        "The last thing to define for our `Seq2SeqTrainer` is how to compute the metrics from the predictions. We need to define a function for this, which will just use the `metric` we loaded earlier, and we have to do a bit of pre-processing to decode the predictions into texts:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UmvbnJ9JIrJd"
      },
      "outputs": [],
      "source": [
        "import nltk\n",
        "import numpy as np\n",
        "\n",
        "def compute_metrics(eval_pred):\n",
        "    predictions, labels = eval_pred\n",
        "    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)\n",
        "    # Replace -100 in the labels as we can't decode them.\n",
        "    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n",
        "    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n",
        "    \n",
        "    # Rouge expects a newline after each sentence\n",
        "    decoded_preds = [\"\\n\".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]\n",
        "    decoded_labels = [\"\\n\".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]\n",
        "    \n",
        "    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)\n",
        "    # Extract a few results\n",
        "    result = {key: value.mid.fmeasure * 100 for key, value in result.items()}\n",
        "    \n",
        "    # Add mean generated length\n",
        "    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]\n",
        "    result[\"gen_len\"] = np.mean(prediction_lens)\n",
        "    \n",
        "    return {k: round(v, 4) for k, v in result.items()}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rXuFTAzDIrJe"
      },
      "source": [
        "Then we just need to pass all of this along with our datasets to the `Seq2SeqTrainer`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "imY1oC3SIrJf"
      },
      "outputs": [],
      "source": [
        "trainer = Seq2SeqTrainer(\n",
        "    model,\n",
        "    args,\n",
        "    train_dataset=tokenized_datasets[\"train\"],\n",
        "    eval_dataset=tokenized_datasets[\"validation\"],\n",
        "    data_collator=data_collator,\n",
        "    tokenizer=tokenizer,\n",
        "    compute_metrics=compute_metrics\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CdzABDVcIrJg"
      },
      "source": [
        "We can now finetune our model by just calling the `train` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uNx5pyRlIrJh",
        "outputId": "077e661e-d36c-469b-89b8-7ff7f73541ec",
        "scrolled": false
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "        </style>\n",
              "      \n",
              "      <progress value='12753' max='12753' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [12753/12753 1:21:48, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "      <th>Rouge1</th>\n",
              "      <th>Rouge2</th>\n",
              "      <th>Rougel</th>\n",
              "      <th>Rougelsum</th>\n",
              "      <th>Gen Len</th>\n",
              "      <th>Runtime</th>\n",
              "      <th>Samples Per Second</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>2.721100</td>\n",
              "      <td>2.479327</td>\n",
              "      <td>28.300900</td>\n",
              "      <td>7.721100</td>\n",
              "      <td>22.243000</td>\n",
              "      <td>22.249600</td>\n",
              "      <td>18.822500</td>\n",
              "      <td>326.333800</td>\n",
              "      <td>34.725000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=12753, training_loss=2.7692033505520146, metrics={'train_runtime': 4909.3835, 'train_samples_per_second': 2.598, 'total_flos': 7.774481450954342e+16, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 335248, 'init_mem_gpu_alloc_delta': 242026496, 'init_mem_cpu_peaked_delta': 18306, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 2637782, 'train_mem_gpu_alloc_delta': 728138240, 'train_mem_cpu_peaked_delta': 138226182, 'train_mem_gpu_peaked_delta': 14677017088})"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aeTOEfN3bqRd"
      },
      "source": [
        "You can now upload the result of the training to the Hub, just execute this instruction:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A0ijFezpbqRd"
      },
      "outputs": [],
      "source": [
        "trainer.push_to_hub()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mPScfd0ubqRd"
      },
      "source": [
        "You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier `\"your-username/the-name-you-picked\"` so for instance:\n",
        "\n",
        "```python\n",
        "from transformers import AutoModelForSeq2SeqLM\n",
        "\n",
        "model = AutoModelForSeq2SeqLM.from_pretrained(\"sgugger/my-awesome-model\")\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ryrWFsczbqRd"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Summarization",
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}