From fbefe1e03e09f151411bdc134784df9d342f68fd Mon Sep 17 00:00:00 2001 From: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com> Date: Tue, 1 Oct 2024 12:13:25 +0200 Subject: [PATCH] modify texts and add illustration Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com> --- notebooks/700_metrics/701a_aupimo.ipynb | 65 +-- notebooks/700_metrics/roc_pro_pimo.svg | 690 ++++++++++++++++++++++++ 2 files changed, 709 insertions(+), 46 deletions(-) create mode 100644 notebooks/700_metrics/roc_pro_pimo.svg diff --git a/notebooks/700_metrics/701a_aupimo.ipynb b/notebooks/700_metrics/701a_aupimo.ipynb index 97135d7081..18fa47df21 100644 --- a/notebooks/700_metrics/701a_aupimo.ipynb +++ b/notebooks/700_metrics/701a_aupimo.ipynb @@ -1,44 +1,41 @@ { "cells": [ { - "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# AUPIMO\n", "\n", - "Basic usage of the AUPIMO metric." + "Basic usage of the metric AUPIMO (pronounced \"a-u-pee-mo\")." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "\n", "# What is AUPIMO?\n", "\n", + "The `Area Under the Per-Image Overlap [curve]` (AUPIMO) is a metric of recall (higher is better) designed for visual anomaly detection.\n", + "\n", + "Inspired by the [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and [PRO](https://link.springer.com/article/10.1007/s11263-020-01400-4) curves, \n", + "\n", + "> AUPIMO is the area under a curve of True Positive Rate (TPR or _recall_) as a function of False Positive Rate (FPR) restricted to a fixed range. \n", "\n", - "The `Area Under the Per-Image Overlap [curve]` (AUPIMO\\*) is a recall\\*\\* metric designed for anomaly segmentation models.\n", + "But:\n", + "- the TPR (Y-axis) is *per-image* (1 image = 1 curve/score);\n", + "- the FPR (X-axis) considers the (average of) **normal** images only; \n", + "- the FPR (X-axis) is in log scale and its range is [1e-5, 1e-4]\\* (harder detection task!).\n", "\n", - "It is inspired by the ROC curve: a curve of False Positive Rate (FPR) vs. True Positive Rate (or _recall_), but:\n", - "- the TPR (Y-axis) is *per-image* (ie. one curve/image);\n", - "- the FPR (X-axis) only considers the (average of) **normal** images; \n", - "- the FPR (X-axis) is in log scale;\n", - "- the FPR (X-axis) range of the area under the curve is [1e-5, 1e-4]\\*\\*\\* (default).\n", + "\\* The score (the area under the curve) is normalized to be in [0, 1].\n", "\n", - "**Interpretation**: “average segmentation recall in an image given that the model (nearly) does not yield false positives in normal images”.\n", + "AUPIMO can be interpreted as\n", "\n", + "> average segmentation recall in an image given that the model (nearly) does not yield false positives in normal images.\n", "\n", - "\\* Pronounced \"a-u-pee-mo\".\n", - "\\*\\* Higher is better.\n", - "\\*\\*\\* The score is normalized to [0, 1].\n", + "References in the last cell.\n", "\n", - "> For more details, check the references below.\n", - "> \n", - "> Paper on arXiv: [arxiv.org/abs/2401.01984](https://arxiv.org/abs/2401.01984) (accepted to BMVC 2024)\n", - "> \n", - "> Medium post: [medium.com/p/c653ac30e802](https://medium.com/p/c653ac30e802)\n", - "> \n", - "> Official repository: [github.com/jpcbertoldo/aupimo](https://github.com/jpcbertoldo/aupimo) (numpy-only and numba-accelerated versions available)" + "![AUROC vs. AUPRO vs. AUPIMO](./roc_pro_pimo.svg)" ] }, { @@ -302,33 +299,9 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "ckpt_path is not provided. Model weights will not be loaded.\n", - "F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead\n", - "Metric `AUPIMO` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "4728dc369c3543b684007c191a369b72", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Predicting: | | 0/? [00:00 + + +image/svg+xmlEach curve summarizesthe test set with di + + +ff + + +erent aggregations. + + +ROC + + +PRO + + +One per image! + + +AUROC + + +AUPRO + + +AUPIMO + + +PIMO + + +i + + +i + + +Recall + +