03_module.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency Distribution"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequency distributions are visual displays that organise and present frequency counts so that the information can be interpreted more easily. Frequency distributions can show absolute frequencies or relative frequencies, such as proportions or percentages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "path = \"data/wnba.csv\"\n",
    "wnba = pd.read_csv(path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate a frequency distribution table using Python, we can use the `Series.value_counts()` method. Let's try it on the Pos column, which describes the position on the court of each individual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "G      60\n",
       "F      33\n",
       "C      25\n",
       "G/F    13\n",
       "F/C    12\n",
       "Name: Pos, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Pos'].value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "188    20\n",
       "193    18\n",
       "175    16\n",
       "185    15\n",
       "183    11\n",
       "173    11\n",
       "191    11\n",
       "196     9\n",
       "178     8\n",
       "180     7\n",
       "170     6\n",
       "198     5\n",
       "201     2\n",
       "168     2\n",
       "206     1\n",
       "165     1\n",
       "Name: Height, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Height'].value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting Frequency distribution tables\n",
    "\n",
    "Pandas sort tables by default inorder of descending frequencies\n",
    "\n",
    "This default is harmless for variables measured on a nominal scale because the unique values, although different, have no direction (we can't say, for instance, that centers are greater or lower than guards). The default actually helps because we can immediately see which values have the greatest or lowest frequencies, we can make comparisons easily, etc.\n",
    "\n",
    "For variables measured on ordinal, interval, or ratio scales, this default makes the analysis of the tables more difficult because the unique values have direction (some unique values are greater or lower than others). Let's consider the table for the Height variable, which is measured on a ratio scale:\n",
    "Consider the distortation of the Height variables above\n",
    "\n",
    "Because the Height variable has direction, we might be interested to find:\n",
    "\n",
    "+ How many players are under 170 cm?\n",
    "+ How many players are very tall (over 185)?\n",
    "+ Are there any players below 160 cm?\n",
    "\n",
    "It's time-consuming to answer these questions using the table above. The solution is to sort the table ourselves."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`wnba['Height'].value_counts()` returns a Series object with the measures of height as indices. This allows us to sort the table by index using the Series.`sort_index()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "165     1\n",
       "168     2\n",
       "170     6\n",
       "173    11\n",
       "175    16\n",
       "178     8\n",
       "180     7\n",
       "183    11\n",
       "185    15\n",
       "188    20\n",
       "191    11\n",
       "193    18\n",
       "196     9\n",
       "198     5\n",
       "201     2\n",
       "206     1\n",
       "Name: Height, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Height'].value_counts().sort_index()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also sort the table by index in descending order using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "206     1\n",
       "201     2\n",
       "198     5\n",
       "196     9\n",
       "193    18\n",
       "191    11\n",
       "188    20\n",
       "185    15\n",
       "183    11\n",
       "180     7\n",
       "178     8\n",
       "175    16\n",
       "173    11\n",
       "170     6\n",
       "168     2\n",
       "165     1\n",
       "Name: Height, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Height'].value_counts().sort_index(ascending = False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Generate a frequency distribution table for the Age variable, which is measured on a ratio scale, and sort the table by unique values.\n",
    "\n",
    "+ Sort the table by unique values in an ascending order, and assign the result to a variable named age_ascending.\n",
    "+ Sort the table by unique values in a descending order, and assign the result to a variable named age_descending.\n",
    "\n",
    "2. Using the variable inspector, analyze one of the frequency distribution tables and brainstorm questions that might be interesting to answer here. These include:\n",
    "\n",
    "+ How many players are under 20?\n",
    "+ How many players are 30 or over?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "age_ascending = wnba['Age'].value_counts().sort_index()\n",
    "age_descending = wnba['Age'].value_counts().sort_index(ascending = False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting Tables for Ordinal Variables"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Series.sort_index()` doesn't work for ordinal variables as it arranges them in alphabetical order instead of their inherent order"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a frequency distribution table for the transformed PTS_ordinal_scale column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_pts_ordinal(row):\n",
    "    if row['PTS'] <= 20:\n",
    "        return 'very few points'\n",
    "    if (20 < row['PTS'] <=  80):\n",
    "        return 'few points'\n",
    "    if (80 < row['PTS'] <=  150):\n",
    "        return 'many, but below average'\n",
    "    if (150 < row['PTS'] <= 300):\n",
    "        return 'average number of points'\n",
    "    if (300 < row['PTS'] <=  450):\n",
    "        return 'more than average'\n",
    "    else:\n",
    "        return 'much more than average'\n",
    "    \n",
    "wnba['PTS_ordinal_scale'] = wnba.apply(make_pts_ordinal, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "very few points             12\n",
       "few points                  27\n",
       "many, but below average     25\n",
       "average number of points    45\n",
       "more than average           21\n",
       "much more than average      13\n",
       "Name: PTS_ordinal_scale, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['PTS_ordinal_scale'].value_counts()[['very few points', 'few points', 'many, but below average', \n",
    "'average number of points','more than average','much more than average']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "very few points             12\n",
       "much more than average      13\n",
       "more than average           21\n",
       "many, but below average     25\n",
       "few points                  27\n",
       "average number of points    45\n",
       "Name: PTS_ordinal_scale, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['PTS_ordinal_scale'].value_counts()[['very few points', 'few points', 'many, but below average', \n",
    "'average number of points','more than average','much more than average']].sort_index(ascending = False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proportions and Percentages"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "long way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "G      0.419580\n",
       "F      0.230769\n",
       "C      0.174825\n",
       "G/F    0.090909\n",
       "F/C    0.083916\n",
       "Name: Pos, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Pos'].value_counts() / len(wnba)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's slightly faster though to use Series.value_counts() with the normalize parameter set to True:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "G      0.419580\n",
       "F      0.230769\n",
       "C      0.174825\n",
       "G/F    0.090909\n",
       "F/C    0.083916\n",
       "Name: Pos, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Pos'].value_counts(normalize = True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find percentages, we just have to multiply the proportions by 100:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "G      41.958042\n",
       "F      23.076923\n",
       "C      17.482517\n",
       "G/F     9.090909\n",
       "F/C     8.391608\n",
       "Name: Pos, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Pos'].value_counts(normalize = True) * 100"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Percentalies and percentale ranks"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The percentile rank of a given score is the percentage of scores in its frequency distribution that are less than that score. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`percentileofscore(a, score, kind='weak')` function from scipy.stats is used to calculate percentale ranks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18.88111888111888"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import percentileofscore\n",
    "percentileofscore(a = wnba['Age'], score = 23, kind = 'weak')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to use kind = 'weak' to indicate that we want to find the percentage of values that are equal to or less than the value we specify in the score parameter."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Percentiles using Pandas"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find percentiles, we can use the Series.describe() method, which returns by default the 25th, the 50th, and the 75th percentiles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    143.000000\n",
       "mean      27.076923\n",
       "std        3.679170\n",
       "min       21.000000\n",
       "25%       24.000000\n",
       "50%       27.000000\n",
       "75%       30.000000\n",
       "max       36.000000\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Age'].describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not interested here in the first three rows of the output (count, mean, and standard deviation). We can use iloc[] to isolate just the output we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "min    21.0\n",
       "25%    24.0\n",
       "50%    27.0\n",
       "75%    30.0\n",
       "max    36.0\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Age'].describe().iloc[3:]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find other percentiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "min      21.0\n",
       "10%      23.0\n",
       "15%      23.0\n",
       "33%      25.0\n",
       "50%      27.0\n",
       "59.2%    28.0\n",
       "85%      31.0\n",
       "90%      32.0\n",
       "max      36.0\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Age'].describe(percentiles = [.1, .15, .33, .5, .592, .85, .9]).iloc[3:]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouped frequency distribution tables"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Used for continous numerical variables\n",
    "\n",
    "bins parameter of `Series.value_counts()` is added for this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(54.941, 60.8]     5\n",
       "(60.8, 66.6]      21\n",
       "(66.6, 72.4]      10\n",
       "(72.4, 78.2]      33\n",
       "(78.2, 84.0]      31\n",
       "(84.0, 89.8]      24\n",
       "(89.8, 95.6]      10\n",
       "(95.6, 101.4]      3\n",
       "(101.4, 107.2]     2\n",
       "(107.2, 113.0]     3\n",
       "Name: Weight, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['Weight'].value_counts(bins = 10).sort_index()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Information Loss on Grouping Data\n",
    "\n",
    "When we increase the number of class intervals, we can get more information, but the table becomes harder to analyze. When we decrease the number of class intervals, we get a boost in comprehensibility, but the amount of information in the table decreases.\n",
    "\n",
    "As a rule of thumb, 10 is a good number of class intervals to choose because it offers a good balance between information and comprehensibility.\n",
    "\n",
    "![Info Loss](img/s1m3_tradeoff.svg)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Readability for Grouped Frequency Tables"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas helps a lot when we need to explore quickly grouped frequency tables. However, the intervals pandas outputs are confusing at first sight:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.417, 99.0]     48\n",
       "(99.0, 196.0]     27\n",
       "(196.0, 293.0]    33\n",
       "(293.0, 390.0]    13\n",
       "(390.0, 487.0]    13\n",
       "(487.0, 584.0]     9\n",
       "Name: PTS, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnba['PTS'].value_counts(bins = 6).sort_index()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above information is not impressive to readers, solved by defining intervals oneself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(0, 100], (100, 200], (200, 300], (300, 400], (400, 500], (500, 600]], dtype='interval[int64, right]')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "intervals = pd.interval_range(start = 0, end = 600, freq = 100)\n",
    "intervals"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we pass the intervals variable to the bins parameter, store the result to gr_freq_table, and print the result, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 100]      49\n",
       "(100, 200]    28\n",
       "(200, 300]    32\n",
       "(300, 400]    17\n",
       "(400, 500]    10\n",
       "(500, 600]     7\n",
       "Name: PTS, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gr_freq_table = wnba[\"PTS\"].value_counts(bins = intervals).sort_index()\n",
    "gr_freq_table"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Apr  5 2022, 06:56:58) \n[GCC 7.5.0]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "d0028ce20b428eb90286e675cfc9d586ae702ba0463c3d3b035a87d9639116bc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}