tutorials/tutorial-statistical-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IR Lab Tutorial: Statistical Analysis\n",
    "\n",
    "This tutorial shows how to conduct a hypothesis test to compare two retrieval approaches.\n",
    "The two runs compared in this example are loaded from the TIRA cache."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Ensure that libraries are imported"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This command loads and starts PyTerrier so that it also works in TIRA.\n",
    "\n",
    "from tira.third_party_integrations import ensure_pyterrier_is_loaded\n",
    "\n",
    "ensure_pyterrier_is_loaded()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PyTerrier must be imported after `ensure_pyterrier_is_loaded` is called.\n",
    "\n",
    "from pyterrier import started, init\n",
    "\n",
    "if not started():\n",
    "    init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IRDSDataset('ir-benchmarks/argsme-touche-2020-task-1-20230209-training')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyterrier import get_dataset\n",
    "\n",
    "dataset = get_dataset('irds:ir-benchmarks/argsme-touche-2020-task-1-20230209-training')\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Create the retrieval pipeline with TIRA\n",
    "\n",
    "In this example, we will just use two existing retrieval components from TIREx: BM25 and DirichletLM, two lexical rankers.\n",
    "We load the approaches via the TIRA API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tira.rest_api_client import Client\n",
    "\n",
    "tira_client = Client()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SourceTransformer()"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dlm = tira_client.pt.from_retriever_submission(\n",
    "    approach='ir-benchmarks/tira-ir-starter/DirichletLM (tira-ir-starter-pyterrier)',\n",
    "    dataset='argsme-touche-2020-task-1-20230209-training',\n",
    ")\n",
    "dlm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SourceTransformer()"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "bm25 = tira_client.pt.from_retriever_submission(\n",
    "    approach='ir-benchmarks/tira-ir-starter/BM25 (tira-ir-starter-pyterrier)',\n",
    "    dataset='argsme-touche-2020-task-1-20230209-training',\n",
    ")\n",
    "bm25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Measure effectiveness\n",
    "\n",
    "Now let us measure the nDCG@10 effectiveness of both systems on the Touché 2020 task 1 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>qid</th>\n",
       "      <th>measure</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DirichletLM</td>\n",
       "      <td>4</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.453743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>BM25</td>\n",
       "      <td>1</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.661871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>BM25</td>\n",
       "      <td>29</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.254807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>BM25</td>\n",
       "      <td>28</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.063621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>DirichletLM</td>\n",
       "      <td>43</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.829042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>DirichletLM</td>\n",
       "      <td>12</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.192790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>BM25</td>\n",
       "      <td>36</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.286423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>DirichletLM</td>\n",
       "      <td>8</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.315163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>BM25</td>\n",
       "      <td>42</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.508146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>BM25</td>\n",
       "      <td>26</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.286346</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name qid      measure     value\n",
       "3   DirichletLM   4  ndcg_cut_10  0.453743\n",
       "49         BM25   1  ndcg_cut_10  0.661871\n",
       "76         BM25  29  ndcg_cut_10  0.254807\n",
       "75         BM25  28  ndcg_cut_10  0.063621\n",
       "41  DirichletLM  43  ndcg_cut_10  0.829042\n",
       "11  DirichletLM  12  ndcg_cut_10  0.192790\n",
       "83         BM25  36  ndcg_cut_10  0.286423\n",
       "7   DirichletLM   8  ndcg_cut_10  0.315163\n",
       "89         BM25  42  ndcg_cut_10  0.508146\n",
       "73         BM25  26  ndcg_cut_10  0.286346"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyterrier.pipelines import Experiment\n",
    "\n",
    "experiment = Experiment(\n",
    "    retr_systems=[\n",
    "        dlm,\n",
    "        bm25,\n",
    "    ],\n",
    "    topics=dataset.get_topics(\"query\"),\n",
    "    qrels=dataset.get_qrels(),\n",
    "    eval_metrics=[\"ndcg_cut_10\"],\n",
    "    names=[\n",
    "        \"DirichletLM\",\n",
    "        \"BM25\",\n",
    "    ],\n",
    "    perquery=True,\n",
    ")\n",
    "experiment.sample(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data frame shows the nDCG@10 values measured for each query and both systems (DrichletLM and BM25). \\\n",
    "So we have pairs of measurements where the same metric (i.e., nDCG@10) is measured using the same input (e.g., query #1) but for two different systems.\n",
    "Let's re-arrange the data frame so that BM25 and DirichletLM values are in separate columns, not rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>measure</th>\n",
       "      <th>value_bm25</th>\n",
       "      <th>value_dlm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.661871</td>\n",
       "      <td>0.880500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.158507</td>\n",
       "      <td>0.633220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.309352</td>\n",
       "      <td>0.752969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.061113</td>\n",
       "      <td>0.192790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.314880</td>\n",
       "      <td>0.434739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>14</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.355866</td>\n",
       "      <td>0.408224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>15</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.094788</td>\n",
       "      <td>0.542364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>16</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.208744</td>\n",
       "      <td>0.443535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>17</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.686715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>18</td>\n",
       "      <td>ndcg_cut_10</td>\n",
       "      <td>0.540948</td>\n",
       "      <td>0.699474</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  qid      measure  value_bm25  value_dlm\n",
       "0   1  ndcg_cut_10    0.661871   0.880500\n",
       "1  10  ndcg_cut_10    0.158507   0.633220\n",
       "2  11  ndcg_cut_10    0.309352   0.752969\n",
       "3  12  ndcg_cut_10    0.061113   0.192790\n",
       "4  13  ndcg_cut_10    0.314880   0.434739\n",
       "5  14  ndcg_cut_10    0.355866   0.408224\n",
       "6  15  ndcg_cut_10    0.094788   0.542364\n",
       "7  16  ndcg_cut_10    0.208744   0.443535\n",
       "8  17  ndcg_cut_10    0.000000   0.686715\n",
       "9  18  ndcg_cut_10    0.540948   0.699474"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_bm25 = experiment[experiment[\"name\"] == \"BM25\"]\\\n",
    "    .drop(columns=[\"name\"])\n",
    "experiment_dlm = experiment[experiment[\"name\"] == \"DirichletLM\"]\\\n",
    "    .drop(columns=[\"name\"])\n",
    "\n",
    "experiment_paired = experiment_bm25.merge(\n",
    "    experiment_dlm,\n",
    "    on=[\"qid\", \"measure\"],\n",
    "    suffixes=(\"_bm25\", \"_dlm\"),\n",
    ")\n",
    "experiment_paired.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Conduct hypothesis tests\n",
    "\n",
    "On this _paired_ measurement data, we can now conduct _paired_ t-tests to test for statistical significance of given hypotheses.\n",
    "Remember that the choice of your test depends (amongst other factors) on how the hypothesis is formulated.\n",
    "\n",
    "Let us test some hypotheses to get a feeling of what this means:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hypothesis 1: BM25 has a significantly different nDCG@10 on Touché 2020 task 1 than DirichletLM.\n",
    "\n",
    "Significance test: two-sided paired t-test \\\n",
    "Significance level: $\\alpha = 0.05$ (i.e., the effect is only considered significant if $p < 0.05$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0865032406710116e-08"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import ttest_rel\n",
    "\n",
    "ttest_rel(\n",
    "    experiment_paired[\"value_bm25\"],\n",
    "    experiment_paired[\"value_dlm\"],\n",
    "    alternative='two-sided',\n",
    ").pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above value is called $p$, the probability of the corresponding null hypothesis (the probability that the effect would be observed by chance). \\\n",
    "Because this is lower than our significance level $\\alpha$, we can reject the null hypothesis and confirm the hypothesis 1. \\\n",
    "Indeed, BM25 and DirichletLM lead to significantly different nDCG@10 scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it would be great to find out which is better. \\\n",
    "One way could be to formulate a hypothesis with a predefined \"direction\". In this example we assume BM25 to be better.\n",
    "\n",
    "#### Hypothesis 2: BM25 has a significantly higher nDCG@10 on Touché 2020 task 1 than DirichletLM.\n",
    "\n",
    "Significance test: one-sided paired t-test \\\n",
    "Significance level: $\\alpha = 0.05$ (or $p < 0.05$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999999945674838"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import ttest_rel\n",
    "\n",
    "ttest_rel(\n",
    "    experiment_paired[\"value_bm25\"],\n",
    "    experiment_paired[\"value_dlm\"],\n",
    "    alternative='greater',\n",
    ").pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, the probability $p$ of the null hypothesis is much higher than our significance level $\\alpha$. \\\n",
    "So we cannot reject the null hypothesis and fail to confirm hypothesis 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, we test the opposite direction: BM25 could be worse w.r.t. nDCG@10 than DirichletLM.\n",
    "\n",
    "#### Hypothesis 2: BM25 has a significantly lower nDCG@10 on Touché 2020 task 1 than DirichletLM.\n",
    "\n",
    "Significance test: one-sided paired t-test \\\n",
    "Significance level: $\\alpha = 0.05$ (or $p < 0.05$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.432516203355058e-09"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import ttest_rel\n",
    "\n",
    "ttest_rel(\n",
    "    experiment_paired[\"value_bm25\"],\n",
    "    experiment_paired[\"value_dlm\"],\n",
    "    alternative='less',\n",
    ").pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, $p$ is less than than our significance level $\\alpha$. We reject the null hypothesis and confirm hypothesis 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}