examples/model_diagnostics/model_diagnostics_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "db768cda",
   "metadata": {},
   "source": [
    "<td>\n",
    "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5611d0",
   "metadata": {},
   "source": [
    "<td>\n",
    "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/model_diagnostics/model_diagnostics_demo.ipynb\" target=\"_blank\"><img\n",
    "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
    "</td>\n",
    "\n",
    "<td>\n",
    "<a href=\"https://github.com/Labelbox/labelbox-python/tree/develop/examples/model_diagnostics/model_diagnostics_demo.ipynb\" target=\"_blank\"><img\n",
    "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attractive-lemon",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Model Diagnostics\n",
    "\n",
    "\n",
    "Throughout the process of training your machine learning (ML) model, you may want to investigate your model's failures in order to understand which areas need improvement. Looking at an error analysis after each training iteration can help you understand whether you need to revise your annotations, make your ontology more clear, or create more training data that targets a specific area.\n",
    "Labelbox now offers a Model Diagnostics tool that analyzes the performance of your model's predictions in a single interface.\n",
    "With Model Diagnostics, you can:\n",
    "*   Inspect model behavior across experiments\n",
    "*   Adjust model hyperparameters and visualize model failures\n",
    "*   Use the Python SDK to create the analysis pipeline\n",
    "\n",
    "## How it works\n",
    "\n",
    "Configuring Model Diagnostics is all done via the SDK. We have created a Google colab notebook to demonstrate this process. The notebook also includes a section that leverages MAL in order to quickly create ground truth annotations.\n",
    "An Experiment is a specific instance of a model generating output in the form of predictions.\n",
    "In Labelbox, the `Model` object represents your ML model and it is what you'll be performing experiments on. It references a set of annotations specified by an ontology. \n",
    "The `Model Run` object represents the experiment itself. It is a specific instance of a `Model` with preconfigured hyperparameters (training data). You can upload inferences across each `Model Run`, filter by IoU score, and compare your model's predictions against the annotations from your training data.\n",
    "\n",
    "## Steps\n",
    "1. Make sure you are signed up for the beta. If not navigate here https://labelbox.com/product/model-diagnostics\n",
    "2. Have a set of ground truth labels in a project\n",
    "3. Install the latest SDK release\n",
    "4. Create a `Model`\n",
    "5. Create a `Model Run`\n",
    "6. Compute predictions\n",
    "7. Compute model performance metrics\n",
    "8. Upload labels, predictions, and metrics\n",
    "9. Navigate to the `Models` tab on Labelbox\n",
    "\n",
    "## Best practices\n",
    "Currently there is a limit of 2000 images per model run. We suggest uploading lower performing examples from your test set.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subsequent-magic",
   "metadata": {
    "id": "subsequent-magic"
   },
   "source": [
    "## Environment Setup\n",
    "\n",
    "Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "voluntary-minister",
   "metadata": {
    "id": "voluntary-minister"
   },
   "outputs": [],
   "source": [
    "!pip install -q \"labelbox[data]\" \\\n",
    "             scikit-image \\\n",
    "             tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "wooden-worship",
   "metadata": {
    "id": "wooden-worship"
   },
   "outputs": [],
   "source": [
    "# Run these if running in a colab notebook\n",
    "COLAB = \"google.colab\" in str(get_ipython())\n",
    "\n",
    "if COLAB:\n",
    "    !git clone https://github.com/Labelbox/labelbox-python.git\n",
    "    !cd labelbox-python\n",
    "    !mv labelbox-python/examples/model_assisted_labeling/*.py ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latter-leone",
   "metadata": {
    "id": "latter-leone"
   },
   "source": [
    "Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "committed-richards",
   "metadata": {
    "id": "committed-richards"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.append('../model_assisted_labeling')\n",
    "\n",
    "import uuid\n",
    "import numpy as np\n",
    "from skimage import measure\n",
    "from tqdm import notebook\n",
    "import requests\n",
    "\n",
    "from labelbox.schema.ontology import OntologyBuilder, Tool\n",
    "from labelbox.data.metrics.group import get_label_pairs\n",
    "from labelbox.data.metrics import feature_miou_metric, feature_confusion_matrix_metric\n",
    "from labelbox import Client, LabelingFrontend, MALPredictionImport\n",
    "from labelbox.data.serialization import NDJsonConverter\n",
    "from labelbox.data.annotation_types import (LabelList, Label, ImageData,\n",
    "                                            MaskData, Mask, Polygon, Point,\n",
    "                                            Rectangle, ObjectAnnotation)\n",
    "\n",
    "try:\n",
    "    from image_model import predict, load_model\n",
    "except ModuleNotFoundError:\n",
    "    # !git clone https://github.com/Labelbox/labelbox-python.git\n",
    "    # !cd labelbox-python && git checkout mea-dev\n",
    "    # !mv labelbox-python/examples/model_assisted_labeling/*.py .\n",
    "    raise Exception(\"You will need to run from the labelbox-python git repo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alternate-promotion",
   "metadata": {
    "id": "alternate-promotion"
   },
   "source": [
    "Configure client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "economic-chase",
   "metadata": {
    "id": "economic-chase"
   },
   "outputs": [],
   "source": [
    "PROJECT_NAME = \"Diagnostics Demo\"\n",
    "MODEL_NAME = \"MSCOCO-Mapillary\"\n",
    "MODEL_VERSION = \"0.0.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c460687d",
   "metadata": {},
   "source": [
    "API Key\n",
    "\n",
    "Provide a valid api key below in order to properly connect to the Labelbox Client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "affecting-myanmar",
   "metadata": {
    "id": "affecting-myanmar"
   },
   "outputs": [],
   "source": [
    "# Add your api key\n",
    "API_KEY = None\n",
    "client = Client(api_key=API_KEY)\n",
    "load_model()  # initialize Tensorflow Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "modern-program",
   "metadata": {
    "id": "modern-program"
   },
   "outputs": [],
   "source": [
    "# Configure for whatever combination of tools and class names that you would like.\n",
    "class_mappings = {\n",
    "    1: {\n",
    "        \"name\": 'person',\n",
    "        \"kind\": Tool.Type.POLYGON\n",
    "    },\n",
    "    2: {\n",
    "        \"name\": 'bicycle',\n",
    "        \"kind\": Tool.Type.SEGMENTATION,\n",
    "        'color': 64\n",
    "    },\n",
    "    3: {\n",
    "        \"name\": 'car',\n",
    "        \"kind\": Tool.Type.BBOX\n",
    "    },\n",
    "    4: {\n",
    "        \"name\": 'motorcycle',\n",
    "        \"kind\": Tool.Type.BBOX\n",
    "    },\n",
    "    6: {\n",
    "        \"name\": 'bus',\n",
    "        \"kind\": Tool.Type.POLYGON\n",
    "    },\n",
    "    7: {\n",
    "        \"name\": 'train',\n",
    "        \"kind\": Tool.Type.POLYGON\n",
    "    },\n",
    "    8: {\n",
    "        \"name\": 'truck',\n",
    "        \"kind\": Tool.Type.POLYGON\n",
    "    },\n",
    "    10: {\n",
    "        \"name\": 'traffic light',\n",
    "        \"kind\": Tool.Type.POINT\n",
    "    },\n",
    "    11: {\n",
    "        \"name\": 'fire hydrant',\n",
    "        \"kind\": Tool.Type.BBOX\n",
    "    },\n",
    "    13: {\n",
    "        \"name\": 'stop sign',\n",
    "        \"kind\": Tool.Type.SEGMENTATION,\n",
    "        'color': 255\n",
    "    },\n",
    "    14: {\n",
    "        \"name\": 'parking meter',\n",
    "        \"kind\": Tool.Type.POINT\n",
    "    },\n",
    "    28: {\n",
    "        \"name\": 'umbrella',\n",
    "        \"kind\": Tool.Type.SEGMENTATION,\n",
    "        'color': 128\n",
    "    },\n",
    "    31: {\n",
    "        \"name\": 'handbag',\n",
    "        \"kind\": Tool.Type.POINT\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dated-burden",
   "metadata": {
    "id": "dated-burden"
   },
   "source": [
    "## Create Predictions\n",
    "* Loop over data_rows, make predictions, and create ndjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "blank-flower",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- setup dataset ---\n",
    "# load mapillary sample\n",
    "sample_csv_url = \"https://raw.githubusercontent.com/Labelbox/labelbox-python/develop/examples/assets/mapillary_sample.csv\"\n",
    "with requests.get(sample_csv_url, stream=True) as r:\n",
    "    image_data = [\n",
    "        row.split(',')\n",
    "        for row in (line.decode('utf-8') for line in r.iter_lines())\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "asian-savings",
   "metadata": {
    "id": "asian-savings"
   },
   "outputs": [],
   "source": [
    "predictions = LabelList()\n",
    "for (image_url, external_id) in notebook.tqdm(image_data[:10]):\n",
    "    image = ImageData(url=image_url, external_id=external_id)\n",
    "    height, width = image.value.shape[:2]\n",
    "    prediction = predict(np.array([image.im_bytes]),\n",
    "                         min_score=0.5,\n",
    "                         height=height,\n",
    "                         width=width)\n",
    "    boxes, classes, seg_masks = prediction[\"boxes\"], prediction[\n",
    "        \"class_indices\"], prediction[\"seg_masks\"]\n",
    "    annotations = []\n",
    "    for box, class_idx, seg in zip(boxes, classes, seg_masks):\n",
    "        if class_idx in class_mappings:\n",
    "            class_info = class_mappings.get(class_idx)\n",
    "            if class_info['kind'] == Tool.Type.POLYGON:\n",
    "                contours = measure.find_contours(seg, 0.5)\n",
    "                pts = contours[0].astype(np.int32)\n",
    "                value = Polygon(points=[\n",
    "                    Point(x=x, y=y) for x, y in np.roll(pts, 1, axis=-1)\n",
    "                ])\n",
    "            elif class_info['kind'] == Tool.Type.BBOX:\n",
    "                value = Rectangle(start=Point(x=box[1], y=box[0]),\n",
    "                                  end=Point(x=box[3], y=box[2]))\n",
    "            elif class_info['kind'] == Tool.Type.POINT:\n",
    "                value = Point(x=(box[1] + box[3]) / 2.,\n",
    "                              y=(box[0] + box[2]) / 2.)\n",
    "            elif class_info['kind'] == Tool.Type.SEGMENTATION:\n",
    "                value = Mask(mask=MaskData.from_2D_arr(seg *\n",
    "                                                       class_info['color']),\n",
    "                             color=(class_info['color'],) * 3)\n",
    "            else:\n",
    "                raise ValueError(\n",
    "                    f\"Unsupported kind found. {class_info['kind']}\")\n",
    "            annotations.append(\n",
    "                ObjectAnnotation(name=class_info['name'], value=value))\n",
    "    predictions.append(Label(data=image, annotations=annotations))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "together-suicide",
   "metadata": {},
   "source": [
    "## Setup a project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "received-nigeria",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- Use the class mapping specified above ( Will include all specified classes )\n",
    "tools = []\n",
    "for target in class_mappings.values():\n",
    "    tools.append(Tool(tool=target['kind'], name=target[\"name\"]))\n",
    "ontology_builder = OntologyBuilder(tools=tools)\n",
    "\n",
    "# --- Optionally Setup ontology from predictions ( Only will include predicted classes )\n",
    "#ontology_builder = predictions.get_ontology()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "stopped-phrase",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Setting up: {PROJECT_NAME}\")\n",
    "\n",
    "project = client.create_project(name=PROJECT_NAME)\n",
    "editor = next(\n",
    "    client.get_labeling_frontends(where=LabelingFrontend.name == \"Editor\"))\n",
    "project.setup(editor, ontology_builder.asdict())\n",
    "\n",
    "dataset = client.create_dataset(name=\"Mapillary Diagnostics Demo\")\n",
    "print(f\"Dataset Created: {dataset.uid}\")\n",
    "project.datasets.connect(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-shelf",
   "metadata": {},
   "source": [
    "## Prepare for upload\n",
    "* Our local annotations need the following:\n",
    "    1. signed url for segmentation masks\n",
    "    2. data rows in labelbox\n",
    "    3. feature schema ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "unavailable-egyptian",
   "metadata": {},
   "outputs": [],
   "source": [
    "signer = lambda _bytes: client.upload_data(content=_bytes, sign=True)\n",
    "predictions.add_url_to_masks(signer) \\\n",
    "         .add_url_to_data(signer) \\\n",
    "         .add_to_dataset(dataset, client.upload_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "perfect-seafood",
   "metadata": {
    "id": "perfect-seafood"
   },
   "source": [
    "## **Optional** - Create labels with [Model Assisted Labeling](https://docs.labelbox.com/en/core-concepts/model-assisted-labeling)\n",
    "\n",
    "* Pre-label image so that we can quickly create ground truth\n",
    "* Create ground truth data for Model Diagnostics\n",
    "* Click on link below to label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "subject-painting",
   "metadata": {
    "id": "subject-painting"
   },
   "outputs": [],
   "source": [
    "RUN_MAL = True\n",
    "if RUN_MAL:\n",
    "    project.enable_model_assisted_labeling()\n",
    "    # Convert from annotation types to import format\n",
    "    ndjson_predictions = NDJsonConverter.serialize(predictions)\n",
    "    upload_task = MALPredictionImport.create_from_objects(\n",
    "        client, project.uid, f'mal-import-{uuid.uuid4()}', ndjson_predictions)\n",
    "    upload_task.wait_until_done()\n",
    "    print(upload_task.state, '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "MV4U1W4H_eMq",
   "metadata": {
    "id": "MV4U1W4H_eMq"
   },
   "outputs": [],
   "source": [
    "print(f\"https://app.labelbox.com/go-label/{project.uid}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stopped-mandate",
   "metadata": {
    "id": "stopped-mandate"
   },
   "source": [
    "## Export Labels\n",
    "\n",
    "We do not support `Skipped` labels and have a limit of **2000**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "excited-seminar",
   "metadata": {
    "id": "excited-seminar"
   },
   "outputs": [],
   "source": [
    "MAX_LABELS = 2000\n",
    "labels = [\n",
    "    l for idx, l in enumerate(project.label_generator()) if idx < MAX_LABELS\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smoking-catering",
   "metadata": {
    "id": "smoking-catering"
   },
   "source": [
    "## Setup Model & Model Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "mental-minnesota",
   "metadata": {
    "id": "mental-minnesota"
   },
   "outputs": [],
   "source": [
    "lb_model = client.create_model(name=MODEL_NAME,\n",
    "                               ontology_id=project.ontology().uid)\n",
    "lb_model_run = lb_model.create_model_run(MODEL_VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cu8h6h0g-Fe2",
   "metadata": {
    "id": "cu8h6h0g-Fe2"
   },
   "source": [
    "Select label ids to upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "static-coordinate",
   "metadata": {
    "id": "static-coordinate"
   },
   "outputs": [],
   "source": [
    "lb_model_run.upsert_labels([label.uid for label in labels])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "g_u1ak2n7qn5",
   "metadata": {
    "id": "g_u1ak2n7qn5"
   },
   "source": [
    "### Compute Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "committed-fairy",
   "metadata": {
    "id": "committed-fairy"
   },
   "outputs": [],
   "source": [
    "pairs = get_label_pairs(labels, predictions, filter_mismatch=True)\n",
    "for (ground_truth, prediction) in pairs.values():\n",
    "    metrics = []\n",
    "    metrics.extend(\n",
    "        feature_miou_metric(ground_truth.annotations, prediction.annotations))\n",
    "    metrics.extend(\n",
    "        feature_confusion_matrix_metric(ground_truth.annotations,\n",
    "                                        prediction.annotations))\n",
    "    prediction.annotations.extend(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "devoted-vatican",
   "metadata": {},
   "source": [
    "### Upload to Labelbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "anonymous-addition",
   "metadata": {
    "id": "anonymous-addition"
   },
   "outputs": [],
   "source": [
    "upload_task = lb_model_run.add_predictions(\n",
    "    f'diagnostics-import-{uuid.uuid4()}',\n",
    "    NDJsonConverter.serialize(predictions))\n",
    "upload_task.wait_until_done()\n",
    "print(upload_task.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uTjGOyIW-3op",
   "metadata": {
    "id": "uTjGOyIW-3op"
   },
   "source": [
    "### Open Model Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "zrll9K6Q9tGK",
   "metadata": {
    "id": "zrll9K6Q9tGK"
   },
   "outputs": [],
   "source": [
    "for idx, model_run_data_row in enumerate(lb_model_run.model_run_data_rows()):\n",
    "    if idx == 5:\n",
    "        break\n",
    "    print(model_run_data_row.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a41f9f33",
   "metadata": {},
   "source": [
    "### Export model run labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4573e2c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_ID = ''\n",
    "MODEL_RUN_ID = ''\n",
    "model = client.get_model(MODEL_ID)\n",
    "\n",
    "model_run = next(filter(lambda run: run['id'] == MODEL_RUN_ID, model.model_runs), None)\n",
    "labels = model_run.export_labels(download=True)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Model Diagnostics Demo",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}