Копия_блокнота_"Модульная_домашняя_работа_ipynb".ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/BroMaArago/MyHomeworkMaximilian/blob/main/%D0%9A%D0%BE%D0%BF%D0%B8%D1%8F_%D0%B1%D0%BB%D0%BE%D0%BA%D0%BD%D0%BE%D1%82%D0%B0_%22%D0%9C%D0%BE%D0%B4%D1%83%D0%BB%D1%8C%D0%BD%D0%B0%D1%8F_%D0%B4%D0%BE%D0%BC%D0%B0%D1%88%D0%BD%D1%8F%D1%8F_%D1%80%D0%B0%D0%B1%D0%BE%D1%82%D0%B0_ipynb%22.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Домашняя работа по темам первого модуля\n",
        "\n",
        "\n",
        "Курс В.И. Фирсановой \"Основы программирования на Python\", НИУ ВШЭ СПб"
      ],
      "metadata": {
        "id": "LKJXB1rGMi22"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Инструкции\n",
        "\n",
        "Вы получите функции, которые нужно дописать. Меткой `### ваш код здесь ###` отмечена строка, в которой вы начинаете работать. Количество строк вашего решения может быть любым в зависимости от условия задачи.\n",
        "\n",
        "Запустите ячейку с кодом, чтобы протестировать свою программу. Ниже вы увидите результат работы своей программы. Если решение верное, вы увидите слово Correct! Если решение нужно доработать, вы увидите слово Improve.\n",
        "\n",
        "Протестируйте свой код и загрузите решение в репозиторий."
      ],
      "metadata": {
        "id": "IzbqCEXsNDAZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Пример работы\n",
        "\n",
        "Так выглядит ваше задание в исходном виде. Условие задачи прописано в docstring - описании функции под строкой с ключевым словом `def`"
      ],
      "metadata": {
        "id": "GLe-QFscN4Xt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(a):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход любую переменную и всегда возвращает число десять\n",
        "\n",
        "    \"\"\"\n",
        "    ### ваш код здесь ###\n",
        "    return a\n",
        "\n",
        "test_data = [7, 'dog', 0.98]\n",
        "test_result = [10, 10, 10]\n",
        "\n",
        "result = [solution(x) for x in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "k--oTJq3MSkC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d02f2e4-620c-49f4-9cd5-30846ec21053"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[7, 'dog', 0.98]\n",
            "Improve\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Образец правильного решения задачи"
      ],
      "metadata": {
        "id": "UKSF2KKvOFjI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5LqVgoADKhka",
        "outputId": "a682bde2-d018-4b69-c5d3-418bdcfa7b8a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[10, 10, 10]\n",
            "Correct!\n"
          ]
        }
      ],
      "source": [
        "def solution(a):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход любую переменную и всегда возвращает число десять\n",
        "\n",
        "    \"\"\"\n",
        "    a = 10\n",
        "    return a\n",
        "\n",
        "test_data = [7, 'dog', 0.98]\n",
        "test_result = [10, 10, 10]\n",
        "\n",
        "result = [solution(x) for x in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Образец неправильного решения задачи"
      ],
      "metadata": {
        "id": "LdP4o0yQOKEH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(a):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход любую переменную и всегда возвращает число десять\n",
        "\n",
        "    \"\"\"\n",
        "    a = 1\n",
        "    return a\n",
        "\n",
        "test_data = [7, 'dog', 0.98]\n",
        "test_result = [10, 10, 10]\n",
        "\n",
        "result = [solution(x) for x in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xpyNLnyGMMtK",
        "outputId": "22776db3-8ece-4463-cb8e-788b835332dd"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1, 1, 1]\n",
            "Improve\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Пишите в тг @vifirsanova, если возникнут вопросы по работе с этим ноутбуком."
      ],
      "metadata": {
        "id": "TQjGjhcmOOu0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Задачи"
      ],
      "metadata": {
        "id": "V7QBQXMlOw9u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 1. Swap variables"
      ],
      "metadata": {
        "id": "5uvHoLGmbuWp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(a, b):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход две переменные (a, b) и меняет местами их значения\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "\n",
        "    buf = a\n",
        "    a = b\n",
        "    b = buf\n",
        "\n",
        "    return a, b\n",
        "\n",
        "test_data = [(5, 10),\n",
        "             ('cat', 'dog'),\n",
        "             ({'name': 'John', 'surname': 'Doe'}, {'name': 'John', 'surname': 'Smith'})]\n",
        "test_result = [(10, 5),\n",
        "               ('dog', 'cat'),\n",
        "               ({'name': 'John', 'surname': 'Smith'}, {'name': 'John', 'surname': 'Doe'})]\n",
        "\n",
        "result = [solution(x, y) for x, y in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "wytx0o4gXavX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fb9d1a7b-6e4f-4efb-a9f6-06968833e400"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[(10, 5), ('dog', 'cat'), ({'name': 'John', 'surname': 'Smith'}, {'name': 'John', 'surname': 'Doe'})]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 2. Имена с большой буквы"
      ],
      "metadata": {
        "id": "MhzTcWyjb1-_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(name):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход имена, записанные в нижнем регистре,\n",
        "    и возвращает те же имена, написанные с большой буквы\n",
        "\n",
        "    \"\"\"\n",
        "    capitalized_name = name.capitalize()\n",
        "\n",
        "    return capitalized_name\n",
        "\n",
        "test_data = ['john', 'mary', 'samantha']\n",
        "test_result = ['John', 'Mary', 'Samantha']\n",
        "\n",
        "result = [solution(name) for name in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "DiYzztjbYtmg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "26e729e7-f256-42bd-bbac-fe429041de94"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['John', 'Mary', 'Samantha']\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 3. Токенизатор"
      ],
      "metadata": {
        "id": "R7bUh5xgcSIo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(sentence):\n",
        "    \"\"\"\n",
        "    Программа принимает текст, записанный в строку, приводит его к нижнему регистру,\n",
        "    разбивает текст на слова и возвращает созданный список слов\n",
        "\n",
        "    \"\"\"\n",
        "    lowercase_sentence = sentence.lower()\n",
        "    tokenized_sentence = lowercase_sentence.split()\n",
        "    return tokenized_sentence\n",
        "\n",
        "test_data = ['This a sample sentence',\n",
        "             'THIS IS A CAPITALIZED SENTENCE',\n",
        "             'and one MORE SENTENCE']\n",
        "test_result = [['this', 'a', 'sample', 'sentence'],\n",
        "               ['this', 'is', 'a', 'capitalized', 'sentence'],\n",
        "               ['and', 'one', 'more', 'sentence']]\n",
        "\n",
        "result = [solution(sentence) for sentence in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "ogthnNS-Z_QA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3c7f816-4249-40de-91ea-33b6c9dca62a"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['this', 'a', 'sample', 'sentence'], ['this', 'is', 'a', 'capitalized', 'sentence'], ['and', 'one', 'more', 'sentence']]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 4. Нативный частотный словарь"
      ],
      "metadata": {
        "id": "mSArNcK2ckXo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(text):\n",
        "    \"\"\"\n",
        "    Программа принимает текст, записанный в строку, приводит его к нижнему регистру,\n",
        "    разбивает текст на слова и создает частотный словарь получившихся токенов.\n",
        "    За один токен принимается словоформа (не нужно приводить слово к начальной форме).\n",
        "    Частотный словарь состоит из пар ключ-значение:\n",
        "    ключ - токен (слово), значение - сколько раз оно встречается в тексте.\n",
        "    Используйте цикл for и условия. Если слово еще не записано в частотный словарь,\n",
        "    внесите его в словарь со значением 1. Если слово уже записано в частотный словарь,\n",
        "    прибавьте к его значению единицу.\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "\n",
        "    text = text.lower()\n",
        "    words = text.split()\n",
        "    freq_dict = {}\n",
        "\n",
        "    for word in words:\n",
        "        if word in freq_dict:\n",
        "            freq_dict[word] += 1\n",
        "        else:\n",
        "            freq_dict[word] = 1\n",
        "\n",
        "    return freq_dict\n",
        "\n",
        "test_data = ['NLP is a subfield of AI and NLP is widely used',\n",
        "             'Count the word frequency and build a frequency dictionary',\n",
        "             'This task is not hard this task is ovewhelming this task is fun']\n",
        "test_result = [{'nlp': 2, 'is': 2, 'a': 1, 'subfield': 1, 'of': 1, 'ai': 1, 'and': 1, 'widely': 1, 'used': 1},\n",
        "               {'count': 1, 'the': 1, 'word': 1, 'frequency': 2, 'and': 1, 'build': 1, 'a': 1, 'dictionary': 1},\n",
        "               {'this': 3, 'task': 3, 'is': 3, 'not': 1, 'hard': 1, 'ovewhelming': 1, 'fun': 1}]\n",
        "\n",
        "result = [solution(text) for text in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "sSFSQoUDbIL_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b179d4ec-1c3e-4c94-b7fd-c558c865dc65"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'nlp': 2, 'is': 2, 'a': 1, 'subfield': 1, 'of': 1, 'ai': 1, 'and': 1, 'widely': 1, 'used': 1}, {'count': 1, 'the': 1, 'word': 1, 'frequency': 2, 'and': 1, 'build': 1, 'a': 1, 'dictionary': 1}, {'this': 3, 'task': 3, 'is': 3, 'not': 1, 'hard': 1, 'ovewhelming': 1, 'fun': 1}]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 5. Найти уникальные словоформы"
      ],
      "metadata": {
        "id": "O68GfLhugw_g"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(text):\n",
        "    \"\"\"\n",
        "    Программа принимает текст, записанный в строку, приводит его к нижнему регистру,\n",
        "    разбивает текст на слова и возвращает список уникальных словоформ.\n",
        "    Какой тип данных помогает отфильтровать объект от повторов?\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    words = text.lower().split()\n",
        "\n",
        "    unique_tokens = set(words)\n",
        "\n",
        "    return unique_tokens\n",
        "\n",
        "\n",
        "test_data = ['NLP is the future of AI and the future is now',\n",
        "             'Count the word frequency to build a frequency dictionary',\n",
        "             'This task is not hard this task is fun']\n",
        "test_result = [{'the', 'is', 'and', 'now', 'of', 'ai', 'nlp', 'future'},\n",
        "               {'the', 'count', 'dictionary', 'frequency', 'a', 'build', 'to', 'word'},\n",
        "               {'hard', 'is', 'task', 'not', 'this', 'fun'}]\n",
        "\n",
        "result = [solution(text) for text in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "hWvEA-5pcjDf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f4319670-9fb3-44f7-eb94-43ed8e9b55ee"
      },
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'future', 'and', 'now', 'nlp', 'is', 'ai', 'the', 'of'}, {'dictionary', 'to', 'a', 'build', 'frequency', 'count', 'the', 'word'}, {'hard', 'task', 'is', 'fun', 'not', 'this'}]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 6. Список покупок"
      ],
      "metadata": {
        "id": "omYck7W4iyq3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(price_list):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход список словарь, в котором записаны наименования\n",
        "    товаров и их стоимость. Программа возвращает человекочитаемый вариант словаря\n",
        "    в виде списка строк вида \"Наименование: $0.00\".\n",
        "    Необходимо произвести итерацию по словарю, с помощью join и форматирования строк\n",
        "    составить строки вида \"Наименование: $0.00\" и сохранить все в единый список.\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    items_list = []\n",
        "    for key, value in price_list.items():\n",
        "      item = ': $'.join([key, format(value, '.2f')])\n",
        "      items_list.append(item)\n",
        "\n",
        "    return items_list\n",
        "\n",
        "test_data = [{'Apples': 2.48, 'Bananas': 1.99, 'Bread': 3.49},\n",
        "             {'Dress': 9.78, 'Pants': 19.96},\n",
        "             {'Ball': 1.33, 'Doll': 1.99, 'Car': 3.50}]\n",
        "test_result = [['Apples: $2.48', 'Bananas: $1.99', 'Bread: $3.49'],\n",
        "               ['Dress: $9.78', 'Pants: $19.96'],\n",
        "               ['Ball: $1.33', 'Doll: $1.99', 'Car: $3.50']]\n",
        "\n",
        "result = [solution(price_list) for price_list in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "sUOFTa98hgCX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "71d8ea65-2642-41e7-e52b-6ba691bad51f"
      },
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['Apples: $2.48', 'Bananas: $1.99', 'Bread: $3.49'], ['Dress: $9.78', 'Pants: $19.96'], ['Ball: $1.33', 'Doll: $1.99', 'Car: $3.50']]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 7. Фильтр слов по длине"
      ],
      "metadata": {
        "id": "6a5HMo-KmjrA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(sentence):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход текст, токенизирует его и возвращает список слов,\n",
        "    длина которых не менее 3 и не более 10 символов. Используйте list comprehension.\n",
        "\n",
        "    \"\"\"\n",
        "    filtered_words = []\n",
        "    sentence = sentence.lower().split()\n",
        "    filtered_words = [word for word in sentence if 3 < len(word) < 10]\n",
        "\n",
        "    return filtered_words\n",
        "\n",
        "test_data = ['This is a sentence',\n",
        "             'Thissentence contains a mistake',\n",
        "             'One more sentence to test_this_code']\n",
        "test_result = [['this', 'sentence'],\n",
        "               ['contains', 'mistake'],\n",
        "               ['more', 'sentence']]\n",
        "\n",
        "result = [solution(sentence) for sentence in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "l7DDGMPApcnn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "76078eea-76f6-445b-9f05-ca5bc638419e"
      },
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['this', 'sentence'], ['contains', 'mistake'], ['more', 'sentence']]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 8. Список гласных"
      ],
      "metadata": {
        "id": "H5Yh3eZimn93"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def solution(text):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход текст и возвращает частотный словарь гласных,\n",
        "    которые встречаются в этом тексте. Используйте list comprehension для создания списка гласных.\n",
        "    Далее с помощью цикла for и условий создайте словарь, в котором ключ - гласная, а значение -\n",
        "    количество повторов этой гласной в тексте. Для этого произведите итерацию по созданному\n",
        "    списку гласных и обновляйте значения словаря, если текущая гласная уже записана в словаре.\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    vowels = ['a', 'e', 'i', 'o', 'u']\n",
        "    vowels_list = [char for char in text.lower() if char in vowels]\n",
        "\n",
        "    vowels_dict = {}\n",
        "    for vowel in vowels_list:\n",
        "        if vowel in vowels_dict:\n",
        "            vowels_dict[vowel] += 1\n",
        "        else:\n",
        "            vowels_dict[vowel] = 1\n",
        "\n",
        "\n",
        "    return vowels_dict\n",
        "\n",
        "test_data = ['Never gonna give you up',\n",
        "             'Never gonna let you down',\n",
        "             'Never gonna run around and desert you']\n",
        "test_result = [{'e': 3, 'o': 2, 'a': 1, 'i': 1, 'u': 2},\n",
        "               {'e': 3, 'o': 3, 'a': 1, 'u': 1},\n",
        "               {'e': 4, 'o': 3, 'a': 3, 'u': 3}]\n",
        "\n",
        "result = [solution(text) for text in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "vecahGvkWjFw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de16b285-8253-49c5-bf9d-5c2397d3f7a0"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'e': 3, 'o': 2, 'a': 1, 'i': 1, 'u': 2}, {'e': 3, 'o': 3, 'a': 1, 'u': 1}, {'e': 4, 'o': 3, 'a': 3, 'u': 3}]\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 9. Удаление стоп-слов"
      ],
      "metadata": {
        "id": "mj4ttl5Lmr7o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "\n",
        "def solution(text):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход текст и возвращает список слов, очищенный от стоп-слов.\n",
        "    Программа записывает в переменную список стоп-слов из nltk,\n",
        "    производит токенизацию с помощью инструмента word_tokenize из nltk,\n",
        "    создает список слов, очищенный от стоп-слов с помощью list comprehension.\n",
        "    Если слово, приведенное к нижнему регистру не обнаружено в списке stop_words,\n",
        "    это слово заносится в список слов, очищенных от стоп-слов (cleaned_text).\n",
        "\n",
        "    \"\"\"\n",
        "    # список стоп-слов из nltk\n",
        "    stop_words = stopwords.words('english')\n",
        "    # токенизация nltk\n",
        "    words = word_tokenize(text)\n",
        "\n",
        "    cleaned_text = [i for i in words if i.lower() not in stop_words]\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    return cleaned_text\n",
        "\n",
        "test_data = ['This is an example sentence with some stop words.',\n",
        "             'I am a tiny sentence with a lot of stop words!',\n",
        "             'Hey, wanna find some stop words?']\n",
        "test_result = [['example', 'sentence', 'stop', 'words', '.'],\n",
        "               ['tiny', 'sentence', 'lot', 'stop', 'words', '!'],\n",
        "               ['Hey', ',', 'wan', 'na', 'find', 'stop', 'words', '?']]\n",
        "\n",
        "result = [solution(text) for text in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "G_w_KXhJnILw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "803ea0fd-c675-4fd2-bac8-67bffc7281d6"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['example', 'sentence', 'stop', 'words', '.'], ['tiny', 'sentence', 'lot', 'stop', 'words', '!'], ['Hey', ',', 'wan', 'na', 'find', 'stop', 'words', '?']]\n",
            "Correct!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "pFrnhURs1q6r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Задача 10. Лемматизация"
      ],
      "metadata": {
        "id": "9BRBSYPanIaI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "nlp = spacy.load(\"en_core_web_sm\")\n",
        "\n",
        "def solution(text):\n",
        "    \"\"\"\n",
        "    Программа принимает на вход текст и возвращает список лемм (начальных форм слова)\n",
        "    с помощью библиотеки Spacy. Результатом работы программы является строка,\n",
        "    в которой слова записаны подряд через пробел.\n",
        "    Текст токенизируется в Spacy, далее с помощью list comprehension программа находит лемму\n",
        "    (token.lemma_) для каждого токена в документе. На месте token может стоять любая другая переменная,\n",
        "    например, i. Затем леммы из списка \"склеиваются\" в одну строку с помощью join.\n",
        "\n",
        "    \"\"\"\n",
        "    # токенизируем текст с помощью Spacy\n",
        "    doc = nlp(text)\n",
        "\n",
        "    lemmatized_text = \" \".join([token.lemma_ for token in doc])\n",
        "\n",
        "\n",
        "    return lemmatized_text\n",
        "\n",
        "test_data = ['they have been to several countries',\n",
        "             'this is the last task hope you\\'re happy',\n",
        "             'happier than ever']\n",
        "test_result = ['they have be to several country',\n",
        "               'this be the last task hope you be happy',\n",
        "               'happy than ever']\n",
        "\n",
        "result = [solution(text) for text in test_data]\n",
        "\n",
        "print(result)\n",
        "\n",
        "if result == test_result:\n",
        "    print('Correct!')\n",
        "else:\n",
        "    print('Improve')"
      ],
      "metadata": {
        "id": "C7-UY9hOnKNw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f2fe34a-f8f2-4327-9487-2303254629d2"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['they have be to several country', 'this be the last task hope you be happy', 'happy than ever']\n",
            "Correct!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "JUYeyO-GZC90"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}