modify texts and add illustration

Signed-off-by: jpcbertoldo <[email protected]>
openvinotoolkit · Oct 1, 2024 · fbefe1e · fbefe1e
1 parent ab3374b
commit fbefe1e
Show file tree

Hide file tree

Showing 2 changed files with 709 additions and 46 deletions.
diff --git a/notebooks/700_metrics/701a_aupimo.ipynb b/notebooks/700_metrics/701a_aupimo.ipynb
@@ -1,44 +1,41 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# AUPIMO\n",
     "\n",
-    "Basic usage of the AUPIMO metric."
+    "Basic usage of the metric AUPIMO (pronounced \"a-u-pee-mo\")."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "\n",
     "# What is AUPIMO?\n",
     "\n",
+    "The `Area Under the Per-Image Overlap [curve]` (AUPIMO) is a metric of recall (higher is better) designed for visual anomaly detection.\n",
+    "\n",
+    "Inspired by the [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and [PRO](https://link.springer.com/article/10.1007/s11263-020-01400-4) curves, \n",
+    "\n",
+    "> AUPIMO is the area under a curve of True Positive Rate (TPR or _recall_) as a function of False Positive Rate (FPR) restricted to a fixed range. \n",
     "\n",
-    "The `Area Under the Per-Image Overlap [curve]` (AUPIMO\\*) is a recall\\*\\* metric designed for anomaly segmentation models.\n",
+    "But:\n",
+    "- the TPR (Y-axis) is *per-image* (1 image = 1 curve/score);\n",
+    "- the FPR (X-axis) considers the (average of) **normal** images only; \n",
+    "- the FPR (X-axis) is in log scale and its range is [1e-5, 1e-4]\\* (harder detection task!).\n",
     "\n",
-    "It is inspired by the ROC curve: a curve of False Positive Rate (FPR) vs. True Positive Rate (or _recall_), but:\n",
-    "- the TPR (Y-axis) is *per-image* (ie. one curve/image);\n",
-    "- the FPR (X-axis) only considers the (average of) **normal** images; \n",
-    "- the FPR (X-axis) is in log scale;\n",
-    "- the FPR (X-axis) range of the area under the curve is [1e-5, 1e-4]\\*\\*\\* (default).\n",
+    "\\* The score (the area under the curve) is normalized to be in [0, 1].\n",
     "\n",
-    "**Interpretation**: “average segmentation recall in an image given that the model (nearly) does not yield false positives in normal images”.\n",
+    "AUPIMO can be interpreted as\n",
     "\n",
+    "> average segmentation recall in an image given that the model (nearly) does not yield false positives in normal images.\n",
     "\n",
-    "\\* Pronounced \"a-u-pee-mo\".\n",
-    "\\*\\* Higher is better.\n",
-    "\\*\\*\\* The score is normalized to [0, 1].\n",
+    "References in the last cell.\n",
     "\n",
-    "> For more details, check the references below.\n",
-    "> \n",
-    "> Paper on arXiv: [arxiv.org/abs/2401.01984](https://arxiv.org/abs/2401.01984) (accepted to BMVC 2024)\n",
-    "> \n",
-    "> Medium post: [medium.com/p/c653ac30e802](https://medium.com/p/c653ac30e802)\n",
-    "> \n",
-    "> Official repository: [github.com/jpcbertoldo/aupimo](https://github.com/jpcbertoldo/aupimo) (numpy-only and numba-accelerated versions available)"
+    "![AUROC vs. AUPRO vs. AUPIMO](./roc_pro_pimo.svg)"
    ]
   },
   {
@@ -302,33 +299,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "ckpt_path is not provided. Model weights will not be loaded.\n",
-      "F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead\n",
-      "Metric `AUPIMO` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "4728dc369c3543b684007c191a369b72",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Predicting: |          | 0/? [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "predictions = engine.predict(dataloaders=datamodule.test_dataloader(), model=model, return_predictions=True)"
    ]
@@ -515,7 +488,7 @@
     "\n",
     "Medium post: [medium.com/p/c653ac30e802](https://medium.com/p/c653ac30e802)\n",
     "\n",
-    "Official repository: [github.com/jpcbertoldo/aupimo](https://github.com/jpcbertoldo/aupimo) (numpy-only and numba-accelerated versions available)\n",
+    "Official repository: [github.com/jpcbertoldo/aupimo](https://github.com/jpcbertoldo/aupimo) (numpy-only API and numba-accelerated versions available)\n",
     "\n",
     "GSoC 2023 page: [summerofcode.withgoogle.com/archive/2023/projects/SPMopugd](https://summerofcode.withgoogle.com/archive/2023/projects/SPMopugd)"
    ]