0_python_intro/Упражнения.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Times New Roman';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"\n",
    "<style>\n",
    "\n",
    "div.cell { /* Tunes the space between cells */\n",
    "margin-top:1em;\n",
    "margin-bottom:1em;\n",
    "}\n",
    "\n",
    "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
    "font-size: 2.2em;\n",
    "line-height:1.4em;\n",
    "text-align:center;\n",
    "}\n",
    "\n",
    "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
    "margin-bottom: -0.4em;\n",
    "}\n",
    "\n",
    "\n",
    "div.text_cell_render { /* Customize text cells */\n",
    "font-family: 'Times New Roman';\n",
    "font-size:1.5em;\n",
    "line-height:1.4em;\n",
    "padding-left:3em;\n",
    "padding-right:3em;\n",
    "}\n",
    "</style>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Упражнения"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NUMPY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/trainings1.jpg'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Импортировать `NumPy` под именем `np`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) Напечатать версию и конфигурацию "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3) Создайте нулевой вектор размером 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4) Создать вектор размера 10, заполненный единицами"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5) Создать вектор размера 10, заполненный нулями, но пятый элемент равен 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6) Создать вектор со значениями от 10 до 49"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7) Создать и потом развернуть вектор (так, чтобы первый элемент стал последним)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8) Создать матрицу (двумерный массив) 3x3 со значениями от 0 до 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9) Создать 3x3 единичную матрицу"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10) Создать массив 3x3x3 со случайными значениями"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11) Создать массив 10x10 со случайными значениями, найти минимум и максимум"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "12) Создать случайный вектор размера 30 и найти среднее значение всех элементов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "13) Создать матрицу с 0 внутри, и 1 на границах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "14) Создать 8x8 матрицу и заполнить её в шахматном порядке, не используя функцию tile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "15) Создать 8x8 матрицу и заполнить её в шахматном порядке, используя функцию tile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "16) Перемножить матрицы размеров 5x3 и 3x2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "17) Дан массив, поменять знак у элементов, значения которых между 3 и 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "18) Создать 5x5 матрицу со значениями в строках от 0 до 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "19) Создать вектор размера 10 со значениями от 0 до 1, не включая ни то, ни другое"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "20) Дан массив 10x2 (точки в декартовой системе координат), преобразовать в полярную"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "21) Заменить максимальный элемент на ноль"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "22) Из двух массивов сделать матрицу Коши $C (C_{ij} = \\frac{1}{x_i - y_j})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "23) Преобразовать массив из float в int"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "24) Случайно расположить p элементов в 2D массив"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "25) Отнять среднее из каждой строки в матрице"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "26) Найти диагональные элементы произведения матриц"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "27) Дан вектор [1, 2, 3, 4, 5], построить новый вектор с тремя нулями между каждым значением"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Разреженные матрицы"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/trainings.png'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Соберите из предложенной таблички csr-матрицу.\n",
    "Дана таблица посещений пользователями сайтов.\n",
    "Строки - описание сессий в интернете для пользователей. Польозватели закодированы числами(1-2-3) в столбце target. Сайты закодированы числами от 1 до 11. Нам не важно, что это были за сайты, при этом 0 - это не сайт, а отсутствие данных, когда за сессию было посещено менее 10 сайтов. \n",
    "Задача - собрать матрицу с частотными описаниями предпочтений пользователей, в которой для каждого сайта указывалось бы, сколько раз пользователь на него вообще заходил за все свои сессии.\n",
    "\n",
    "Насколько эффективнее sparse матрица будет при количестве разных сайтов 10000? При количестве пользователей 1000?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('data/train_data_3users.csv')\n",
    "data.drop('session_id', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) Создать 2 случайные матрицы размерами 10х10 с долей ненулевых элементов не более 10% и перемножить в их в csr и numpy формате. Сравнить время "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PANDAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/trainings2.jpg'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**В задании предлагается с помощью Pandas ответить на несколько вопросов по данным репозитория UCI [Adult](https://archive.ics.uci.edu/ml/datasets/Adult) (качать данные не надо – они уже есть в репозитории)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Уникальные значения признаков (больше информации по ссылке выше):\n",
    "- age: continuous.\n",
    "- workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.\n",
    "- fnlwgt: continuous.\n",
    "- education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.\n",
    "- education-num: continuous.\n",
    "- marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.\n",
    "- occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.\n",
    "- relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.\n",
    "- race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.\n",
    "- sex: Female, Male.\n",
    "- capital-gain: continuous.\n",
    "- capital-loss: continuous.\n",
    "- hours-per-week: continuous.\n",
    "- native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.   \n",
    "- salary: >50K,<=50K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# чтоб картинки рисовались в тетрадке\n",
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "plt.rcParams['figure.figsize'] = (10, 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education-num</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         workclass  fnlwgt  education  education-num  \\\n",
       "0   39         State-gov   77516  Bachelors             13   \n",
       "1   50  Self-emp-not-inc   83311  Bachelors             13   \n",
       "2   38           Private  215646    HS-grad              9   \n",
       "3   53           Private  234721       11th              7   \n",
       "4   28           Private  338409  Bachelors             13   \n",
       "\n",
       "       marital-status         occupation   relationship   race     sex  \\\n",
       "0       Never-married       Adm-clerical  Not-in-family  White    Male   \n",
       "1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n",
       "2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n",
       "3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n",
       "4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n",
       "\n",
       "   capital-gain  capital-loss  hours-per-week native-country salary  \n",
       "0          2174             0              40  United-States  <=50K  \n",
       "1             0             0              13  United-States  <=50K  \n",
       "2             0             0              40  United-States  <=50K  \n",
       "3             0             0              40  United-States  <=50K  \n",
       "4             0             0              40           Cuba  <=50K  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('data/adult.data.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Сколько мужчин и женщин (признак *sex*) представлено в этом наборе данных?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) Каков средний возраст (признак *age*) женщин?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3) Постройте гистограмму распределения (*bar plot*) образования людей (признак *education*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4) Каковы средние значения и среднеквадратичные отклонения возраста тех, кто получает более 50K в год (признак *salary*) и тех, кто получает менее 50K в год?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5) Правда ли, что люди, которые получают больше 50k, имеют как минимум высшее образование? (признак *education - Bachelors, Prof-school, Assoc-acdm, Assoc-voc, Masters* или *Doctorate*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6) Среди кого больше доля зарабатывающих много (>50K): среди женатых или холостых мужчин (признак *marital-status*)? Женатыми считаем тех, у кого *marital-status* начинается с *Married* (Married-civ-spouse, Married-spouse-absent или Married-AF-spouse), остальных считаем холостыми."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7) Посчитайте среднее время работы (*hours-per-week*) зарабатывающих мало и много (*salary*) для каждой страны (*native-country*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Графики"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/trainings.JPG' width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Загрузка и знакомство с данными\n",
    "\n",
    "Для работы вам понадобятся данные из учебного конкурса на kaggle [«Прогноз популярности статьи на Хабре»](https://inclass.kaggle.com/c/howpop-habrahabr-favs-lognorm). Файл **`howpop_train.csv`** уже лежит в нашем репозитории в каталоге **`data`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/howpop_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(134137, 17)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>url</th>\n",
       "      <td>https://habrahabr.ru/post/18284/</td>\n",
       "      <td>https://habrahabr.ru/post/18285/</td>\n",
       "      <td>https://habrahabr.ru/post/18286/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>domain</th>\n",
       "      <td>habrahabr.ru</td>\n",
       "      <td>habrahabr.ru</td>\n",
       "      <td>habrahabr.ru</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>post_id</th>\n",
       "      <td>18284</td>\n",
       "      <td>18285</td>\n",
       "      <td>18286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>published</th>\n",
       "      <td>2008-01-01 18:19:00</td>\n",
       "      <td>2008-01-01 18:30:00</td>\n",
       "      <td>2008-01-01 18:34:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <td>@Tapac</td>\n",
       "      <td>@DezmASter</td>\n",
       "      <td>@DezmASter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flow</th>\n",
       "      <td>develop</td>\n",
       "      <td>design</td>\n",
       "      <td>design</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>polling</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>content_len</th>\n",
       "      <td>4305</td>\n",
       "      <td>7344</td>\n",
       "      <td>8431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>Новогодний подарок блоггерам — WordPress 2.3.2</td>\n",
       "      <td>Сумасшедшие яйца, или сервис для отслеживания ...</td>\n",
       "      <td>Сумасшедшие яйца, или сервис для отслеживания ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comments</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>favs</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>views</th>\n",
       "      <td>236</td>\n",
       "      <td>353</td>\n",
       "      <td>1200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>votes_plus</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>votes_minus</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>views_lognorm</th>\n",
       "      <td>-0.792687</td>\n",
       "      <td>-0.485214</td>\n",
       "      <td>0.44609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>favs_lognorm</th>\n",
       "      <td>-1.34407</td>\n",
       "      <td>-0.831946</td>\n",
       "      <td>1.79402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comments_lognorm</th>\n",
       "      <td>-2.43687</td>\n",
       "      <td>-1.87319</td>\n",
       "      <td>0.574656</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               0  \\\n",
       "url                             https://habrahabr.ru/post/18284/   \n",
       "domain                                              habrahabr.ru   \n",
       "post_id                                                    18284   \n",
       "published                                    2008-01-01 18:19:00   \n",
       "author                                                    @Tapac   \n",
       "flow                                                     develop   \n",
       "polling                                                    False   \n",
       "content_len                                                 4305   \n",
       "title             Новогодний подарок блоггерам — WordPress 2.3.2   \n",
       "comments                                                       0   \n",
       "favs                                                           0   \n",
       "views                                                        236   \n",
       "votes_plus                                                     0   \n",
       "votes_minus                                                    0   \n",
       "views_lognorm                                          -0.792687   \n",
       "favs_lognorm                                            -1.34407   \n",
       "comments_lognorm                                        -2.43687   \n",
       "\n",
       "                                                                  1  \\\n",
       "url                                https://habrahabr.ru/post/18285/   \n",
       "domain                                                 habrahabr.ru   \n",
       "post_id                                                       18285   \n",
       "published                                       2008-01-01 18:30:00   \n",
       "author                                                   @DezmASter   \n",
       "flow                                                         design   \n",
       "polling                                                       False   \n",
       "content_len                                                    7344   \n",
       "title             Сумасшедшие яйца, или сервис для отслеживания ...   \n",
       "comments                                                          1   \n",
       "favs                                                              1   \n",
       "views                                                           353   \n",
       "votes_plus                                                        1   \n",
       "votes_minus                                                       0   \n",
       "views_lognorm                                             -0.485214   \n",
       "favs_lognorm                                              -0.831946   \n",
       "comments_lognorm                                           -1.87319   \n",
       "\n",
       "                                                                  2  \n",
       "url                                https://habrahabr.ru/post/18286/  \n",
       "domain                                                 habrahabr.ru  \n",
       "post_id                                                       18286  \n",
       "published                                       2008-01-01 18:34:00  \n",
       "author                                                   @DezmASter  \n",
       "flow                                                         design  \n",
       "polling                                                       False  \n",
       "content_len                                                    8431  \n",
       "title             Сумасшедшие яйца, или сервис для отслеживания ...  \n",
       "comments                                                         47  \n",
       "favs                                                             72  \n",
       "views                                                          1200  \n",
       "votes_plus                                                       45  \n",
       "votes_minus                                                       5  \n",
       "views_lognorm                                               0.44609  \n",
       "favs_lognorm                                                1.79402  \n",
       "comments_lognorm                                           0.574656  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Избавимся сразу от переменных, названия которых заканчиваются на `_lognorm`. Выберем их с помощью `filter()` и удалим `drop`-ом:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.drop(filter(lambda c: c.endswith('_lognorm'), df.columns), \n",
    "        axis = 1,       # axis = 1: столбцы \n",
    "        inplace = True) # избавляет от необходимости сохранять датасет"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>post_id</th>\n",
       "      <td>134137.0</td>\n",
       "      <td>181307.054265</td>\n",
       "      <td>81766.350702</td>\n",
       "      <td>18284.0</td>\n",
       "      <td>115565.0</td>\n",
       "      <td>182666.0</td>\n",
       "      <td>257401.0</td>\n",
       "      <td>314088.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>content_len</th>\n",
       "      <td>134137.0</td>\n",
       "      <td>7516.929699</td>\n",
       "      <td>8724.771640</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2179.0</td>\n",
       "      <td>4949.0</td>\n",
       "      <td>9858.0</td>\n",
       "      <td>352471.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comments</th>\n",
       "      <td>134137.0</td>\n",
       "      <td>39.625994</td>\n",
       "      <td>62.398958</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>2266.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>favs</th>\n",
       "      <td>134137.0</td>\n",
       "      <td>71.999866</td>\n",
       "      <td>145.854135</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>8718.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>views</th>\n",
       "      <td>134137.0</td>\n",
       "      <td>16631.013084</td>\n",
       "      <td>31479.819691</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2100.0</td>\n",
       "      <td>7600.0</td>\n",
       "      <td>18700.0</td>\n",
       "      <td>1730000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>votes_plus</th>\n",
       "      <td>133566.0</td>\n",
       "      <td>35.536888</td>\n",
       "      <td>42.461073</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>1466.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>votes_minus</th>\n",
       "      <td>133566.0</td>\n",
       "      <td>8.050035</td>\n",
       "      <td>11.398282</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>445.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                count           mean           std      min       25%  \\\n",
       "post_id      134137.0  181307.054265  81766.350702  18284.0  115565.0   \n",
       "content_len  134137.0    7516.929699   8724.771640      1.0    2179.0   \n",
       "comments     134137.0      39.625994     62.398958      0.0       7.0   \n",
       "favs         134137.0      71.999866    145.854135      0.0       8.0   \n",
       "views        134137.0   16631.013084  31479.819691      3.0    2100.0   \n",
       "votes_plus   133566.0      35.536888     42.461073      0.0      11.0   \n",
       "votes_minus  133566.0       8.050035     11.398282      0.0       2.0   \n",
       "\n",
       "                  50%       75%        max  \n",
       "post_id      182666.0  257401.0   314088.0  \n",
       "content_len    4949.0    9858.0   352471.0  \n",
       "comments         19.0      48.0     2266.0  \n",
       "favs             27.0      78.0     8718.0  \n",
       "views          7600.0   18700.0  1730000.0  \n",
       "votes_plus       22.0      45.0     1466.0  \n",
       "votes_minus       5.0      10.0      445.0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>url</th>\n",
       "      <td>134137</td>\n",
       "      <td>134137</td>\n",
       "      <td>https://habrahabr.ru/company/mailru/blog/270741/</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>domain</th>\n",
       "      <td>134137</td>\n",
       "      <td>2</td>\n",
       "      <td>habrahabr.ru</td>\n",
       "      <td>97048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>published</th>\n",
       "      <td>134137</td>\n",
       "      <td>130291</td>\n",
       "      <td>2011-06-14 15:52:00</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <td>97657</td>\n",
       "      <td>22077</td>\n",
       "      <td>@alizar</td>\n",
       "      <td>5292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flow</th>\n",
       "      <td>97048</td>\n",
       "      <td>6</td>\n",
       "      <td>develop</td>\n",
       "      <td>53318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>polling</th>\n",
       "      <td>134137</td>\n",
       "      <td>2</td>\n",
       "      <td>False</td>\n",
       "      <td>129412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>134137</td>\n",
       "      <td>133628</td>\n",
       "      <td>Интересные события, произошедшие в выходные</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            count  unique                                               top  \\\n",
       "url        134137  134137  https://habrahabr.ru/company/mailru/blog/270741/   \n",
       "domain     134137       2                                      habrahabr.ru   \n",
       "published  134137  130291                               2011-06-14 15:52:00   \n",
       "author      97657   22077                                           @alizar   \n",
       "flow        97048       6                                           develop   \n",
       "polling    134137       2                                             False   \n",
       "title      134137  133628       Интересные события, произошедшие в выходные   \n",
       "\n",
       "             freq  \n",
       "url             1  \n",
       "domain      97048  \n",
       "published      39  \n",
       "author       5292  \n",
       "flow        53318  \n",
       "polling    129412  \n",
       "title          56  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe(include = ['object', 'bool'] # бинарные и категориальные переменные\n",
    "           ).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# настройка внешнего вида графиков в seaborn\n",
    "sns.set_style(\"dark\")\n",
    "sns.set_palette(\"RdBu\")\n",
    "sns.set_context(\"notebook\", font_scale = 1.5, \n",
    "                rc = { \"figure.figsize\" : (15, 5), \"axes.titlesize\" : 18 })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Столбец **`published`** (время публикации) содержит строки. Чтобы мы могли работать с этими данными как с датой/временем публикации, приведём их к типу `datetime`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "object\n",
      "datetime64[ns]\n"
     ]
    }
   ],
   "source": [
    "print(df.published.dtype)\n",
    "df['published'] = pd.to_datetime(df.published, yearfirst = True)\n",
    "print(df.published.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим несколько столбцов на основе данных о времени публикации:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['year'] = [d.year for d in df.published]\n",
    "df['month'] = [d.month for d in df.published]\n",
    "\n",
    "df['dayofweek'] = [d.isoweekday() for d in df.published]\n",
    "df['hour'] = [d.hour for d in df.published]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "Теперь Ваша очередь. В каждом пункте предлагается построить картинку и с ее помощью ответить на вопрос. Конечно, можно попытаться ответить на все вопросы только с Pandas, без картинок, но мы будем смотреть на (красивые) визуализации.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1\\. В каком месяце (и какого года) было больше всего публикаций?\n",
    "\n",
    "* март 2016,\n",
    "* март 2015,\n",
    "* апрель 2015,\n",
    "* апрель 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2\\. Проанализируйте публикации в этом месяце (из вопроса 1)\n",
    "\n",
    "Выберите один или несколько вариантов:\n",
    "\n",
    "* На графике не заметны какие-либо выбросы или аномалии,\n",
    "* Один или несколько дней сильно выделяются из общей картины,\n",
    "* На хабре _всегда_ больше статей, чем на гиктаймсе,\n",
    "* По субботам на гиктаймс и на хабрахабр публикуют примерно одинаковое число статей.\n",
    "\n",
    "Подсказки: постройте график зависимости числа публикаций от дня; используйте параметр `hue`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3\\. Выберите верное утверждение\n",
    "\n",
    "* Больше всего просмотров (в среднем) набирают статьи, опубликованные в 12 часов дня;\n",
    "* У опубликованных в 10 утра постов больше всего комментариев  (в среднем);\n",
    "* Больше всего просмотров  (в среднем) набирают статьи, опубликованные в 6 часов утра;\n",
    "* Максимальное число комментариев на гиктаймсе набрала статья, опубликованная в 9 часов вечера;\n",
    "* На хабре дневные статьи комментируют чаще  (в среднем), чем вечерние."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4\\. Кого из топ-20 авторов (по числу статей) чаще всего минусуют (в среднем)?\n",
    "\n",
    "* @Mordatyj\n",
    "* @Mithgol\n",
    "* @alizar\n",
    "* @ilya42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5\\. Сравните субботы и понедельники\n",
    "\n",
    "Правда ли, что по субботам авторы пишут в основном днём, а по понедельникам — в основном вечером?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}