Goodreads Data Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e74b620",
   "metadata": {},
   "source": [
    "# Goodreads Data Analysis\n",
    "\n",
    "## I. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fe6ec8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import itertools\n",
    "import datapane as dp\n",
    "import plotly.graph_objects as go\n",
    "import plotly.figure_factory as ff\n",
    "import plotly.express as px"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886606ff",
   "metadata": {},
   "source": [
    "## II. Load the data\n",
    "\n",
    "Here we'll load the csv file and separate its columns into two dataframes. The first one will keep the details about the books that we won't use in the analysis, and the second one will store the data we need for the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "997c8cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "csv = pd.read_csv('./GoodReads_100k_books.csv')\n",
    "csv.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72edd685",
   "metadata": {},
   "outputs": [],
   "source": [
    "to_analyse = ['title', 'bookformat', 'pages', 'author', 'genre', 'reviews', 'totalratings']\n",
    "details = ['title', 'desc', 'rating', 'img', 'link']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28666d99",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_book_details = csv.filter(items=details, axis=1)\n",
    "df_book_details.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67a54f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = csv.filter(items=to_analyse, axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73e21eb",
   "metadata": {},
   "source": [
    "## III. Preparation\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1771157",
   "metadata": {},
   "source": [
    "### Basic Information\n",
    "\n",
    "Firstly we'll print some basic information about the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ecf142b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3d46527",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbcfce34",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['bookformat'].nunique() # Unique values in bookformat column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba91b4b5",
   "metadata": {},
   "source": [
    "Before we can count how many unique values are there in the author and genre columns, we'll have to split the values stored as strings in these columns for each row of the dataframe. Only then we'll be able to treat each author or genre category independently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ae28c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "split_if_not_missing = lambda w: w if pd.isnull(w) else w.split(',')\n",
    "\n",
    "df['genre'] = df['genre'].apply(split_if_not_missing)\n",
    "df['author'] = df['author'].apply(split_if_not_missing)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "604dd35d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def unique_categories(data_frame, column, sort_ascending=False):\n",
    "    '''\n",
    "    Todo: write document string.\n",
    "    '''\n",
    "    column_notnull_values = data_frame[data_frame[column].notnull()][column]\n",
    "    df_cat = pd.DataFrame(\n",
    "        {\n",
    "            'count': pd.Series(itertools.chain.from_iterable(column_notnull_values)).value_counts()\n",
    "        }\n",
    "    ).sort_values(by='count', ascending=sort_ascending)\n",
    "    return df_cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de3542b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_categories(df, 'author').shape[0] # Number of authors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49da53c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_categories(df, 'genre').shape[0] # Number of genres"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7587152",
   "metadata": {},
   "source": [
    "### Missing Values\n",
    "\n",
    "Now that we already know some basic information about the books dataset, we should search for missing values in order to have them properly treated before the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edc462d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53f7c787",
   "metadata": {},
   "source": [
    "There are no missing values for most variables we are going to use, which is great! However there are some missing values in the bookformat and genre columns. We should also look for rows with 0 in the pages column, since any real book has more than 0 pages, and thus it's more appropriate to treat these cases of 0 pages as missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b929b9a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.replace({'pages': {0: np.nan}})\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a095cd68",
   "metadata": {},
   "source": [
    "There are too many different bookformats, but most of them might actually represent a very small part of the books in our dataset. Let's select only the most representative bookformats and filter the dataframe. Depending on the criteria we chose to select the bookformats, this may also remove all the rows where the format is missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4cfb055",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cumulative_percentage(data_frame, column, ascending=False, sufix='cumulative_pc') -> pd.DataFrame:\n",
    "    '''\n",
    "    Computes the cumulative percentage for a specified numeric column of a data_frame.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    data_frame: DataFrame\n",
    "    column: str\n",
    "    ascending: bool, default False\n",
    "        Sort ascending vs descending.\n",
    "    sufix: str, default 'cumulative_pc'\n",
    "        String to append to the end of the cumulative percentage column name.\n",
    "        Sufix is preceded by a underscore by default.\n",
    "        \n",
    "    Returns:\n",
    "        DataFrame: A copy of input data_frame with an aditional column for the cumulative percentage.\n",
    "    '''\n",
    "    \n",
    "    result_df = data_frame.sort_values(by=column, ascending=ascending).reset_index(drop=True)\n",
    "    result_df[column + '_' + sufix] = result_df[column].cumsum() / result_df[column].sum()\n",
    "    return result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e72c2ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_by_representativeness(data_frame, column, mode=1, threshold=0.95, threshold_column=None) -> pd.DataFrame:\n",
    "    '''\n",
    "    Selects the most or the least representative categories from column of a data_frame.\n",
    "\n",
    "    The data is grouped by the column values to measure the value counts or the sum of values from another column, if specified in threshold_column.\n",
    "    After sorting the data by the measured values, only those that correspond to a fraction smaller than or equal to the threshold will be selected.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    data_frame: DataFrame\n",
    "    column: str\n",
    "        The name of the column from which the categories will be selected.\n",
    "    mode: 0 or 1, default 1\n",
    "        Whether to select the most (1) or the least (0) representative categories.\n",
    "    threshold: {0 to 1}, default 0.95\n",
    "        Proportion of categories measured values to be selected. If 0, an empty list is returned. If 1, all categories are returned.\n",
    "    threshold_column str, optional:\n",
    "        The name of a column with numeric values to be used as measure of representativeness.\n",
    "        \n",
    "    Returns:\n",
    "        DataFrame: A DataFrame with the selected categories from column.\n",
    "    '''\n",
    "    if threshold_column is None:\n",
    "        agg_func = 'count'\n",
    "        agg_col_name = agg_func\n",
    "        grouped = data_frame.groupby(column).size().reset_index(name=agg_col_name)\n",
    "    else:\n",
    "        agg_func = 'sum'\n",
    "        agg_col_name = threshold_column + '_' + agg_func\n",
    "        grouped = data_frame.groupby(column).agg(\n",
    "            {threshold_column: agg_func}\n",
    "        ).rename(\n",
    "            {threshold_column: agg_col_name},\n",
    "            axis=1\n",
    "        ).reset_index()\n",
    "    \n",
    "    ascending = not bool(mode)\n",
    "    sufix = 'cumulative_pc'\n",
    "    grouped = cumulative_percentage(\n",
    "        data_frame=grouped,\n",
    "        column=agg_col_name,\n",
    "        ascending=ascending,\n",
    "        sufix=sufix\n",
    "    )\n",
    "    selected = grouped[grouped[agg_col_name + '_' + sufix] <= threshold]\n",
    "    \n",
    "    return selected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfec8003",
   "metadata": {},
   "outputs": [],
   "source": [
    "representative_bookformats = select_by_representativeness(df, 'bookformat', threshold=0.979)\n",
    "representative_bookformats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de47f1d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['bookformat'].isin(representative_bookformats['bookformat'])]\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a25e109",
   "metadata": {},
   "source": [
    "Now that only the genre and pages columns still have missing values, we should decide wheter to drop the rows, to input the values, or to let the missing values in the dataframe. Since for this project we won't build a ML model and the data in these columns will be used in different steps of the analysis, we are going to drop the rows, but only those where both the genre and the number of pages are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0817552d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.dropna(subset=['genre', 'pages'], how='all').reset_index(drop=True)\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eea2178",
   "metadata": {},
   "source": [
    "### Additional columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea955989",
   "metadata": {},
   "source": [
    "Before we can start our analysis, let's create bins for the number of pages of the books to be able to deal with categories instead of having to deal with the numeric variable itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21e01e33",
   "metadata": {},
   "outputs": [],
   "source": [
    "bins=list(range(0,1000,100))+[max(df['pages'])]\n",
    "\n",
    "pages_labels=[]\n",
    "for i in range(len(bins)-1):\n",
    "    if i == len(bins)-2:\n",
    "        pages_labels.append('Over ' + str(bins[i]))\n",
    "        continue\n",
    "    pages_labels.append(str(bins[i]) + \" to \" + str(bins[i+1]))\n",
    "\n",
    "pages_categories = pd.cut(\n",
    "    x=df['pages'],\n",
    "    bins=bins,\n",
    "    labels=pages_labels\n",
    ")\n",
    "df.insert(2, 'pages_cat', pages_categories)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f0965a",
   "metadata": {},
   "source": [
    "Finally, we need a measure of how much the readers got engaged by the book, so let's add a column named 'readers_engagement' for this feature. We will define it as the sum of reviews and total ratings received by a book on Goodreads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bb4785a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['readers_engagement'] = df['reviews'] + df['totalratings']\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad584064",
   "metadata": {},
   "source": [
    "## IV. Analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "503a1b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we can see how is the shape of the distribution of the books by number of pages and bookformat.\n",
    "# It's clearly a skewed distribution.\n",
    "fig = px.histogram(\n",
    "    data_frame=df,\n",
    "    x='pages',\n",
    "    color='bookformat',\n",
    "    category_orders=dict(\n",
    "        bookformat=representative_bookformats['bookformat']\n",
    "    ),\n",
    "    barmode='overlay'\n",
    ")\n",
    "\n",
    "fig.update_traces(\n",
    "    xbins={'start': 0, 'end': 5000, 'size': 100}\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd0d582f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Distribution of the books by readers engagement values\n",
    "# Also skewed. Should apply log scale!\n",
    "fig = px.histogram(df, 'readers_engagement')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358df295",
   "metadata": {},
   "source": [
    "### 1. How does the book size and the book format affect the readers engagement?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59c77151",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the median engagement for each book size (pages) interval\n",
    "pages_vs_engagement = df.groupby('pages_cat').agg({'readers_engagement': 'median'}).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a595a7ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the readers engagement for books in each category of size and format\n",
    "\n",
    "fig = px.box(\n",
    "    data_frame=df,\n",
    "    x='pages_cat',\n",
    "    y='readers_engagement',\n",
    "    color='bookformat',\n",
    "    category_orders=dict(\n",
    "        pages_cat=pages_labels,\n",
    "        bookformat=representative_bookformats['bookformat']\n",
    "    ),\n",
    "    log_y=True,\n",
    "    labels={\n",
    "        'pages_cat': 'Book size (pages)',\n",
    "        'readers_engagement': 'Readers engagement',\n",
    "        'bookformat': 'Book Format'\n",
    "    },\n",
    "    title='Readers engagement by book size and format',\n",
    "    hover_name='title'\n",
    ")\n",
    "\n",
    "fig.add_trace(go.Scatter(\n",
    "        x=pages_vs_engagement['pages_cat'],\n",
    "        y=pages_vs_engagement['readers_engagement'],\n",
    "        mode='markers',\n",
    "        marker_symbol=41,\n",
    "        marker_color='black',\n",
    "        marker_line_width=2,\n",
    "        marker_size=40,\n",
    "        name='Median Engagement',\n",
    "        hovertemplate=\n",
    "        'Pages: %{x}<br>'+\n",
    "        'Median Engagement: %{y}'\n",
    "    )\n",
    ")\n",
    "\n",
    "fig.update_layout(\n",
    "    height=700,\n",
    "    width=1000,\n",
    "    legend=dict(\n",
    "        orientation='h',\n",
    "        yanchor='bottom',\n",
    "        y=1.01,\n",
    "        xanchor='left',\n",
    "        x=0\n",
    "    )\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "852ba313",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Publish to datapane\n",
    "report = dp.Report(dp.Plot(fig))\n",
    "report.upload(name='book_popularity_vs_size_and_format', open=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73485e08",
   "metadata": {},
   "source": [
    "## 2. How to make a popular genre combination?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f927b0df",
   "metadata": {},
   "source": [
    "Let's first prepare a new dataframe with unique genres and count how many times they appear in our original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9234785a",
   "metadata": {},
   "outputs": [],
   "source": [
    "genres = unique_categories(df, 'genre').sort_values(by='count', ascending=False)\n",
    "genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46cd34b0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig = px.line(\n",
    "    data_frame=genres,\n",
    "    x=genres.index,\n",
    "    y='count',\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "562f1b32",
   "metadata": {},
   "source": [
    "We will also need an auxiliary dataframe where each colum corresponds to a genre and the values are dummies (0 or 1) that indicate whether a book has been classified with this genre or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f88506",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cat_dummies_from_lists(cat_lists, unique_cats):\n",
    "    '''\n",
    "    Searches each list in cat_lists for the categoric values specified in unique_cats.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    cat_lists: Series\n",
    "        A series of lists containing categoric values.\n",
    "    unique_cats: list\n",
    "        The categories (unique values) to search in each list from cat_lists.\n",
    "    \n",
    "    Returns: DataFrame\n",
    "        A DataFrame where each row corresponds to a list from cat_lists and each column corresponds to a category.\n",
    "        The value is 1 if the category was found in that list and 0 if it wasn't.\n",
    "    '''\n",
    "    dummies = {}\n",
    "    for cat in unique_cats:\n",
    "        dummies[cat] = cat_lists.apply(lambda x: x if x is np.nan else int(cat in x))\n",
    "    return pd.DataFrame(dummies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97cf3fba",
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_dummies = cat_dummies_from_lists(df['genre'], genres.index.to_list())\n",
    "genre_dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4f7f23",
   "metadata": {},
   "source": [
    "### 2.1. Which are the most and the least popular book genres?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d24be9d",
   "metadata": {},
   "source": [
    "We will first store the sum of readers engagement grouped by the books genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "502f8bd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_engagements = genre_dummies.mul(df['readers_engagement'], axis=0)\n",
    "genres['total_engagement'] = genre_engagements.sum()\n",
    "genres['avg_engagement'] = genre_engagements.mean()\n",
    "genres['dev_from_mean'] = genres['avg_engagement'] - genres['avg_engagement'].mean()\n",
    "genres = genres.sort_values(by='avg_engagement', ascending=False).reset_index().rename({'index': 'genre'}, axis=1)\n",
    "genres"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5452ce26",
   "metadata": {},
   "source": [
    "Now we can rank any number of most popular genres. Let's list the top 10 genres with the highest average readers engagement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ae37717",
   "metadata": {},
   "outputs": [],
   "source": [
    "popular_n = 10\n",
    "\n",
    "genres.head(popular_n)['genre']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decd80f9",
   "metadata": {},
   "source": [
    "In the previous step, 'audiobook' has showed up as one of the top 10 popularity genres. However 'audiobook' would fit better as a category for the books format, so we shouldn't consider it as a valid book genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261f8ed2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove \"Audiobook\" from genres\n",
    "genres = genres[genres['genre'] != 'Audiobook']\n",
    "genre_dummies = genre_dummies.drop(columns=['Audiobook'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7392a626",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the 50 most popular genres\n",
    "popular_n = 50\n",
    "\n",
    "fig = px.bar(\n",
    "    data_frame=genres.head(popular_n),\n",
    "    x='avg_engagement',\n",
    "    y='genre',\n",
    "    orientation='h',\n",
    "    labels={'genre': 'Genre', 'avg_engagement': 'Average readers engagement', 'total_engagement': 'Total engagement'},\n",
    "    title='The ' + str(popular_n) + ' most popular genres',\n",
    "    hover_name='genre',\n",
    "    hover_data={\n",
    "        'genre': False,\n",
    "        'total_engagement': ':,',\n",
    "        'avg_engagement': ':.2f'\n",
    "    }\n",
    ")\n",
    "\n",
    "fig.update_layout(\n",
    "    height=900,\n",
    "    width=900,\n",
    "    margin={'pad': 10},\n",
    "    bargap=0.5,\n",
    "    yaxis={'categoryorder': 'total ascending', 'tickmode': 'linear'}\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa195b23",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Plot the complete popularity rank\n",
    "\n",
    "fig = px.histogram(\n",
    "    data_frame=genres,\n",
    "    y='genre',\n",
    "    x='dev_from_mean',\n",
    "    orientation='h',\n",
    "    title='Overview of genre popularity'\n",
    ")\n",
    "\n",
    "color = ['SeaGreen' if i < 50 else 'DodgerBlue' if val >= 0 else 'Tomato' for i, val in enumerate(genres['dev_from_mean'])]\n",
    "\n",
    "fig.update_traces(\n",
    "    marker=dict(color=color),\n",
    "    customdata=np.stack((genres.index+1, genres['avg_engagement']), axis=-1),\n",
    "    hovertemplate=\n",
    "    '<b>#%{customdata[0]}: %{y}</b><br>' +\n",
    "    '<br>Popularity (average readers engagement): %{customdata[1]:,.2f}' +\n",
    "    '<br>Popularity (deviation from mean): %{x:.2f}' +\n",
    "    '<extra></extra>'\n",
    ")\n",
    "\n",
    "fig.update_layout(\n",
    "    height=800,\n",
    "    showlegend=False,\n",
    "    xaxis={\n",
    "        'title': 'Deviation from mean'\n",
    "    },\n",
    "    yaxis={\n",
    "        'title': 'Genre',\n",
    "        'autorange': 'reversed'\n",
    "    }\n",
    ")\n",
    "\n",
    "\n",
    "fig.add_hrect(\n",
    "    y0=genres['genre'].iloc[50],\n",
    "    y1=genres['genre'].iloc[0],\n",
    "    line_width=0,\n",
    "    fillcolor='PaleGreen',\n",
    "    opacity=0.6,\n",
    "    layer='below',\n",
    "    annotation={\n",
    "        'text': 'Top 50 popular genres',\n",
    "        'y': genres['genre'].iloc[25],\n",
    "        'yanchor': 'middle',\n",
    "        'ay': 0,\n",
    "        'x': 0.5,\n",
    "        'xanchor': 'center',\n",
    "    }\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00b629a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Publish to datapane\n",
    "report = dp.Report(dp.Plot(fig))\n",
    "report.upload(name='overview_of_genres_popularity', open=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b902043",
   "metadata": {},
   "source": [
    "### 2.2. How are genres associated to each other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a54b87a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute correlation between all genres\n",
    "genres_corr = genre_dummies.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a09963",
   "metadata": {},
   "source": [
    "The correlation between genres will be ploted as a heatmap, but there are too many categories to include in a single heatmap. So let's filter the genres_corr dataframe to get a better visualization of the most popular genres and their relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f25d2096",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Leave only the genres with above average readers engagement\n",
    "corr_filtered_df = genres_corr.filter(genres[genres['dev_from_mean'] >= 0]['genre'], axis = 1)\n",
    "# Leave only the genres with at least one correlation greater than 0.15 (or lower than -0.15)\n",
    "corr_filtered_df = corr_filtered_df.where(lambda x: abs(x) >= 0.15).dropna(how='all')\n",
    "include = corr_filtered_df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3fc565b",
   "metadata": {},
   "outputs": [],
   "source": [
    "genres_corr_filtered = genres_corr.filter(include, axis=1).filter(include, axis=0)\n",
    "genre_dummies_filtered = genre_dummies.filter(include, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abe24236",
   "metadata": {},
   "source": [
    "In order to visualize more clearly the existing relations between genres, we will clusterize the correlations dataframe before plotting the heatmap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5ac736d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.cluster.hierarchy import linkage, fcluster\n",
    "from scipy.spatial.distance import squareform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cf6771c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dissimilarity = 1 - abs(genres_corr_filtered)\n",
    "Z = linkage(squareform(dissimilarity), 'complete')\n",
    "\n",
    "# Clusterize the data\n",
    "threshold = 0.90\n",
    "labels = fcluster(Z, threshold, criterion='distance')\n",
    "\n",
    "# Keep the indices to sort labels\n",
    "labels_order = np.argsort(labels)\n",
    "\n",
    "# Build a new dataframe with the sorted columns\n",
    "for idx, i in enumerate(genre_dummies_filtered.columns[labels_order]):\n",
    "    if idx == 0:\n",
    "        clustered = pd.DataFrame(genre_dummies_filtered[i])\n",
    "    else:\n",
    "        df_to_append = pd.DataFrame(genre_dummies_filtered[i])\n",
    "        clustered = pd.concat([clustered, df_to_append], axis=1)\n",
    "\n",
    "clustered_corr = clustered.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e93d75b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the heatmap\n",
    "\n",
    "fig = px.imshow(\n",
    "    img=round(clustered_corr,2),\n",
    "    color_continuous_scale='Balance_r',\n",
    "    range_color=[-1,1],\n",
    "    title='Genres correlation'\n",
    ")\n",
    "\n",
    "fig.update_layout(\n",
    "    height=900,\n",
    "    width=900,\n",
    "    coloraxis=dict(\n",
    "        colorbar=dict(\n",
    "            title='Correlation'\n",
    "        )\n",
    "    ),\n",
    "    xaxis=dict(\n",
    "        showspikes=True,\n",
    "        spikemode='across'\n",
    "    ),\n",
    "    yaxis=dict(\n",
    "        showspikes=True,\n",
    "        spikemode='across'\n",
    "    ),\n",
    ")\n",
    "\n",
    "dist_threshold = 7\n",
    "added_annotations_loc = []\n",
    "for i, item in enumerate(genres.head(50)['genre']):\n",
    "    loc = clustered_corr.columns.get_loc(item)\n",
    "    dist = abs(np.subtract(added_annotations_loc, loc))\n",
    "    if not any(dist < dist_threshold):\n",
    "        added_annotations_loc.append(loc)\n",
    "        fig.add_annotation(\n",
    "            text=f'#{i+1}: {item}',\n",
    "            font=dict(\n",
    "                size=9\n",
    "            ),\n",
    "            x=item,\n",
    "            xanchor='left' if loc < clustered_corr.shape[1]/2 else 'right',\n",
    "            ax=35 if loc < clustered_corr.shape[1]/2 else -35,\n",
    "            y=item,\n",
    "            yanchor='middle',\n",
    "            ay=0,\n",
    "            showarrow=True\n",
    "        )\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ee1dc16",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Publish to datapane\n",
    "report = dp.Report(dp.Plot(fig))\n",
    "report.upload(name='genres_correlation', open=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7107c46",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}