[AC][Smartlab] Action recognition and object detection models and acc…

…uracy tools (openvinotoolkit#3107) * added smartlab models * fixed * fixed * added smartlab models * fixed readme * fixed readme * fixed readme * fixed * fixed yml * removed dead link * removed redundant space * replaces assets images * replaces assets images * fixed * fixed image name * fixed readme and ymls * fixed readme and ymls * fixed * fixed readme * fixed object detection output format * fixed object detection output format * drop smartlab-action-recognition * fixed index.md * fixed accurach-check.yml * fixed accurach-check.yml * fixed * fixed * Revert "fixed" This reverts commit e3e80c1. * fixed * Update README.md * Update accuracy-check.yml * Update README.md * Update accuracy-check.yml * Update README.md * Update README.md * fixed
yangwang201911 · Feb 21, 2022 · 0590913 · 0590913
1 parent 00d8a33
commit 0590913
Show file tree

Hide file tree

Showing 14 changed files with 359 additions and 0 deletions.
diff --git a/data/dataset_definitions.yml b/data/dataset_definitions.yml
@@ -1461,3 +1461,35 @@ datasets:
       data_dir: annotation
       input_suffix: in
       reference_suffix: out
+
+  - name: smartlab_detection_10cl_top
+    data_source: object_detection/streams_1/top/images
+    annotation_conversion:
+      converter: mscoco_detection
+      annotation_file: object_detection/streams_1/top/annotations/instances_glb1cls10.json
+    annotation: mscoco_detection_top_10cls.pickle
+    dataset_meta: mscoco_detection_top_10cls.json
+
+  - name: smartlab_detection_3cl_top
+    data_source: object_detection/streams_1/top/images
+    annotation_conversion:
+      converter: mscoco_detection
+      annotation_file: object_detection/streams_1/top/annotations/instances_glb2bcls3.json
+    annotation: mscoco_detection_top_3cls.pickle
+    dataset_meta: mscoco_detection_top_3cls.json
+
+  - name: smartlab_detection_10cl_high
+    data_source: object_detection/streams_1/high/images
+    annotation_conversion:
+      converter: mscoco_detection
+      annotation_file: object_detection/streams_1/high/annotations/instances_glb1cls10.json
+    annotation: mscoco_detection_high_10cls.pickle
+    dataset_meta: mscoco_detection_high_10cls.json
+
+  - name: smartlab_detection_3cl_high
+    data_source: object_detection/streams_1/high/images
+    annotation_conversion:
+      converter: mscoco_detection
+      annotation_file: object_detection/streams_1/high/annotations/instances_glb2bcls3.json
+    annotation: mscoco_detection_high_3cls.pickle
+    dataset_meta: mscoco_detection_high_3cls.json
diff --git a/models/intel/index.md b/models/intel/index.md
@@ -73,6 +73,10 @@ network to detect objects of the same type better.
 | [yolo-v2-tiny-ava-sparse-30-0001](./yolo-v2-tiny-ava-sparse-30-0001/README.md)          | 6.975 | 15.12  |
 | [yolo-v2-tiny-ava-sparse-60-0001](./yolo-v2-tiny-ava-sparse-60-0001/README.md)          | 6.975 | 15.12  |
 | [yolo-v2-tiny-vehicle-detection-0001](./yolo-v2-tiny-vehicle-detection-0001/README.md)  | 5.424 | 11.229 |
+| [smartlab-object-detection-0001](./smartlab-object-detection-0001/README.md)            | 1.077 | 0.8908 |
+| [smartlab-object-detection-0002](./smartlab-object-detection-0002/README.md)            | 1.073 | 0.8894 |
+| [smartlab-object-detection-0003](./smartlab-object-detection-0003/README.md)            | 1.077 | 0.8908 |
+| [smartlab-object-detection-0004](./smartlab-object-detection-0004/README.md)            | 1.073 | 0.8894 |
 
 ## Object Recognition Models
 

diff --git a/models/intel/smartlab-object-detection-0001/README.md b/models/intel/smartlab-object-detection-0001/README.md
@@ -0,0 +1,49 @@
+# smartlab-object-detection-0001
+
+## Use Case and High-Level Description
+
+This is a smartlab object detector that is based on YoloX for 416x416 resolution.
+
+## Example
+
+![](./assets/frame0001_top1.jpg)
+
+## Specification
+
+Accuracy metrics obtained on Smartlab validation dataset with yolox adapter for converted model.
+
+| Metric                          | Value                                     |
+|---------------------------------|-------------------------------------------|
+| [COCO mAP (0.5:0.05:0.95)]      | 20.33%                                    |
+| GFlops                          | 1.077                                     |
+| MParams                         | 0.8908                                    |
+| Source framework                | PyTorch\*                                 |
+
+## Inputs
+
+Image, name: `images`, shape: `1, 3, 416, 416` in the format `B, C, H, W`, where:
+
+- `B` - batch size
+- `C` - number of channels
+- `H` - image height
+- `W` - image width
+
+Expected color order is `BGR`.
+
+## Outputs
+
+The array of detection summary info, name - `output`, shape - `1, 3549, 15`, format is `B, N, 15`, where:
+
+- `B` - batch size
+- `N` - number of detection boxes
+
+Detection box has format [`x`, `y`, `h`, `w`, `box_score`, `class_no_1`, ..., `class_no_10`], where:
+
+- (`x`, `y`) - raw coordinates of box center
+- `h`, `w` - raw height and width of box
+- `box_score` - confidence of detection box
+- `class_no_1`, ..., `class_no_10` - probability distribution over the classes in logits format.
+
+## Legal Information
+
+[*] Other names and brands may be claimed as the property of others.
diff --git a/models/intel/smartlab-object-detection-0001/accuracy-check.yml b/models/intel/smartlab-object-detection-0001/accuracy-check.yml
@@ -0,0 +1,31 @@
+models:
+  - name: smartlab-object-detection-0001
+
+    launchers:
+      - framework: openvino
+        adapter:
+          type: yolox
+          anchors: yolox
+          classes: 10
+
+    datasets:
+      - name: smartlab_detection_10cl_top
+
+        preprocessing:
+          - type: resize
+            size: 416
+
+        postprocessing:
+          - type: resize_prediction_boxes
+            rescale: True
+          - type: nms
+            overlap: 0.3
+          - type: clip_boxes
+            apply_to: prediction
+
+        metrics:
+          - name: [email protected]:0.05:95
+            type: coco_precision
+            max_detections: 100
+            threshold: '0.5:0.05:0.95'
+            reference: 0.2033
diff --git a/models/intel/smartlab-object-detection-0001/assets/frame0001_top1.jpg b/models/intel/smartlab-object-detection-0001/assets/frame0001_top1.jpg
diff --git a/models/intel/smartlab-object-detection-0002/README.md b/models/intel/smartlab-object-detection-0002/README.md
@@ -0,0 +1,48 @@
+# smartlab-object-detection-0002
+
+## Use Case and High-Level Description
+
+This is a smartlab object detector that is based on YoloX-nano for 416x416 resolution.
+
+## Example
+
+![](./assets/frame0001_top2.jpg)
+
+## Specification
+
+Accuracy metrics obtained on Smartlab validation dataset with yolox adapter for converted model.
+
+| Metric                          | Value                                     |
+| [COCO mAP (0.5:0.05:0.95)]      | 6.06%                                     |
+| GFlops                          | 1.073                                     |
+| MParams                         | 0.8894                                    |
+| Source framework                | PyTorch\*                                 |
+
+## Inputs
+
+Image, name: `images`, shape: `1, 3, 416, 416` in the format `B, C, H, W`, where:
+
+- `B` - batch size
+- `C` - number of channels
+- `H` - image height
+- `W` - image width
+
+Expected color order is `BGR`.
+
+## Outputs
+
+The array of detection summary info, name - `output`, shape - `1, 3549, 8`, format is `B, N, 8`, where:
+
+- `B` - batch size
+- `N` - number of detection boxes
+
+Detection box has format [`x`, `y`, `h`, `w`, `box_score`, `class_no_1`, ...,`class_no_3`], where:
+
+- (`x`, `y`) - raw coordinates of box center
+- `h`, `w` - raw height and width of box
+- `box_score` - confidence of detection box
+- `class_no_1`, ..., `class_no_3` - probability distribution over the classes in logits format.
+
+## Legal Information
+
+[*] Other names and brands may be claimed as the property of others.
diff --git a/models/intel/smartlab-object-detection-0002/accuracy-check.yml b/models/intel/smartlab-object-detection-0002/accuracy-check.yml
@@ -0,0 +1,31 @@
+models:
+  - name: smartlab-object-detection-0002
+
+    launchers:
+      - framework: openvino
+        adapter:
+          type: yolox
+          anchors: yolox
+          classes: 3
+
+    datasets:
+      - name: smartlab_detection_3cl_top
+
+        preprocessing:
+          - type: resize
+            size: 416
+
+        postprocessing:
+          - type: resize_prediction_boxes
+            rescale: True
+          - type: nms
+            overlap: 0.2
+          - type: clip_boxes
+            apply_to: prediction
+
+        metrics:
+          - name: [email protected]:0.05:95
+            type: coco_precision
+            max_detections: 100
+            threshold: '0.5:0.05:0.95'
+            reference: 0.0606
diff --git a/models/intel/smartlab-object-detection-0002/assets/frame0001_top2.jpg b/models/intel/smartlab-object-detection-0002/assets/frame0001_top2.jpg
diff --git a/models/intel/smartlab-object-detection-0003/README.md b/models/intel/smartlab-object-detection-0003/README.md
@@ -0,0 +1,49 @@
+# smartlab-object-detection-0003
+
+## Use Case and High-Level Description
+
+This is a smartlab object detector that is based on YoloX for 416x416 resolution.
+
+## Example
+
+![](./assets/frame0001_front1.jpg)
+
+## Specification
+
+Accuracy metrics obtained on Smartlab validation dataset with yolox adapter for converted model.
+
+| Metric                          | Value                                     |
+|---------------------------------|-------------------------------------------|
+| [COCO mAP (0.5:0.05:0.95)]      | 30.38%                                    |
+| GFlops                          | 1.077                                     |
+| MParams                         | 0.8908                                    |
+| Source framework                | PyTorch\*                                 |
+
+## Inputs
+
+Image, name: `images`, shape: `1, 3, 416, 416` in the format `B, C, H, W`, where:
+
+- `B` - batch size
+- `C` - number of channels
+- `H` - image height
+- `W` - image width
+
+Expected color order is `BGR`.
+
+## Outputs
+
+The array of detection summary info, name - `output`, shape - `1, 3549, 15`, format is `B, N, 15`, where:
+
+- `B` - batch size
+- `N` - number of detection boxes
+
+Detection box has format [`x`, `y`, `h`, `w`, `box_score`, `class_no_1`, ..., `class_no_10`], where:
+
+- (`x`, `y`) - raw coordinates of box center
+- `h`, `w` - raw height and width of box
+- `box_score` - confidence of detection box
+- `class_no_1`, ..., `class_no_10` - probability distribution over the classes in logits format.
+
+## Legal Information
+
+[*] Other names and brands may be claimed as the property of others.
diff --git a/models/intel/smartlab-object-detection-0003/accuracy-check.yml b/models/intel/smartlab-object-detection-0003/accuracy-check.yml
@@ -0,0 +1,31 @@
+models:
+  - name: smartlab-object-detection-0003
+
+    launchers:
+      - framework: openvino
+        adapter:
+          type: yolox
+          anchors: yolox
+          classes: 10
+
+    datasets:
+      - name: smartlab_detection_10cl_high
+
+        preprocessing:
+          - type: resize
+            size: 416
+
+        postprocessing:
+          - type: resize_prediction_boxes
+            rescale: True
+          - type: nms
+            overlap: 0.3
+          - type: clip_boxes
+            apply_to: prediction
+
+        metrics:
+          - name: [email protected]:0.05:95
+            type: coco_precision
+            max_detections: 100
+            threshold: '0.5:0.05:0.95'
+            reference: 0.3038
diff --git a/models/intel/smartlab-object-detection-0003/assets/frame0001_front1.jpg b/models/intel/smartlab-object-detection-0003/assets/frame0001_front1.jpg
diff --git a/models/intel/smartlab-object-detection-0004/README.md b/models/intel/smartlab-object-detection-0004/README.md
@@ -0,0 +1,53 @@
+# smartlab-object-detection-0004
+
+## Use Case and High-Level Description
+
+This is a smartlab object detector that is based on YoloX for 416x416 resolution.
+
+## Example
+
+![](./assets/frame0001_front2.jpg)
+
+## Specification
+
+Accuracy metrics obtained on Smartlab validation dataset with yolox adapter for converted model.
+
+| Metric                          | Value                                     |
+|---------------------------------|-------------------------------------------|
+| [COCO mAP (0.5:0.05:0.95)]      | 11.18%                                    |
+| GFlops                          | 1.073                                     |
+| MParams                         | 0.8894                                    |
+| Source framework                | PyTorch\*                                 |
+
+Average Precision (AP) is defined as an area under
+the [precision/recall](https://en.wikipedia.org/wiki/Precision_and_recall)
+curve.
+
+## Inputs
+
+Image, name: `images`, shape: `1, 3, 416, 416` in the format `B, C, H, W`, where:
+
+- `B` - batch size
+- `C` - number of channels
+- `H` - image height
+- `W` - image width
+
+Expected color order is `BGR`.
+
+## Outputs
+
+The array of detection summary info, name - `output`, shape - `1, 3549, 8`, format is `B, N, 8`, where:
+
+- `B` - batch size
+- `N` - number of detection boxes
+
+Detection box has format [`x`, `y`, `h`, `w`, `box_score`, `class_no_1`, ..., `class_no_3`], where:
+
+- (`x`, `y`) - raw coordinates of box center
+- `h`, `w` - raw height and width of box
+- `box_score` - confidence of detection box
+- `class_no_1`, ..., `class_no_3` - probability distribution over the classes in logits format.
+
+## Legal Information
+
+[*] Other names and brands may be claimed as the property of others.
diff --git a/models/intel/smartlab-object-detection-0004/accuracy-check.yml b/models/intel/smartlab-object-detection-0004/accuracy-check.yml
@@ -0,0 +1,31 @@
+models:
+  - name: smartlab-object-detection-0004
+
+    launchers:
+      - framework: openvino
+        adapter:
+          type: yolox
+          anchors: yolox
+          classes: 3
+
+    datasets:
+      - name: smartlab_detection_3cl_high
+
+        preprocessing:
+          - type: resize
+            size: 416
+
+        postprocessing:
+          - type: resize_prediction_boxes
+            rescale: True
+          - type: nms
+            overlap: 0.2
+          - type: clip_boxes
+            apply_to: prediction
+
+        metrics:
+          - name: [email protected]:0.05:95
+            type: coco_precision
+            max_detections: 100
+            threshold: '0.5:0.05:0.95'
+            reference: 0.1118
diff --git a/models/intel/smartlab-object-detection-0004/assets/frame0001_front2.jpg b/models/intel/smartlab-object-detection-0004/assets/frame0001_front2.jpg