OpenVino Blindspot Assistance

Table of Contents

• 1. Introduction
• 2. Bussines Logic
• 3. Prerequisites
  • 3.1. Hardware
  • 3.2. Software
  • 3.3. Checks
• 4. Building
  • 4.1. Basic Build
  • 4.2. Build with Docker
    • 4.2.1. Prerequisites
    • 4.2.2. Installation
  • 4.3. Build in DockerHub
  • 4.4. MQTT Broker Installation
• 5. Usage
  • 5.1. Run
    • 5.1.1. Examples
    • 5.1.2. Other models
    • 5.1.3. Run in Docker
• 6. Troubleshooting
• 7. [Optional] AWS (In Progress)

1. Introduction

This project consists on showcasing the advantages of the Intel’s OpenVINO toolkit. We will develop a Blindspot Assistance case scenario, where we will define a blindspot area to detect vehicles and pedestrians, up to 4 cameras simultaneously. For that, we will use the OpenVINO toolkit and OpenCV.

2. Bussines Logic

Using OpenVino’s model detection we can easily detect pedestrians and vehicles with great accuracy. We are currently using a pedestrian and vehicle detection model that is included with OpenVino out-of-the-box:

1. pedestrian-and-vehicle-detector-adas-0001

3. Prerequisites

To run the application in this tutorial, the OpenVINO™ toolkit (Release R2 or greater) and its dependencies must already be installed and verified using the included demos. Installation instructions may be found at: https://software.intel.com/en-us/articles/OpenVINO-Install-Linux

If to be used, any optional hardware must also be installed and verified including:

• USB camera - Standard USB Video Class (UVC) camera.

• Intel® Core™ CPU with integrated graphics.

• VPU - USB Intel® Movidius™ Neural Compute Stick and what is being referred to as "Myriad"

A summary of what is needed:

3.1. Hardware

• Target and development platforms meeting the requirements described in the "System Requirements" section of the OpenVINO™ toolkit documentation which may be found at: https://software.intel.com/en-us/openvino-toolkit

Note: While writing this tutorial, an Intel® i7-8550U with Intel® HD graphics 520 GPU was used as both the development and target platform.

• Optional:

  • Intel® Movidius™ Neural Compute Stick

  • USB UVC camera

  • Intel® Core™ CPU with integrated graphics.

3.2. Software

• OpenVINO™ toolkit supported Linux operating system. This tutorial was run on 64-bit Ubuntu 18.04.3 LTS updated to kernel 4.15.0-66 following the OpenVINO™ toolkit installation instructions.

• The latest OpenVINO™ toolkit installed and verified. Supported versions +2019 R2. (Lastest version supported 2019 R3.1).

• Git(git) for downloading from the GitHub repository.

3.3. Checks

By now you should have completed the Linux installation guide for the OpenVINO™ toolkit, however before continuing, please ensure:

• That after installing the OpenVINO™ toolkit you have run the supplied demo samples

• If you have and intend to use a GPU: You have installed and tested the GPU drivers

• If you have and intend to use a USB camera: You have connected and tested the USB camera

• If you have and intend to use a Myriad: You have connected and tested the USB Intel® Movidius™ Neural Compute Stick

• That your development platform is connected to a network and has Internet access. To download all the files for this tutorial, you will need to access GitHub on the Internet.

4. Building

4.1. Basic Build

1. Clone the repository at desired location:

git clone https://github.com/incluit/OpenVino-Blindspot-Assistance.git

2. The first step is to configure the build environment for the OpenCV toolkit by sourcing the "setupvars.sh" script:

source  /opt/intel/openvino/bin/setupvars.sh

3. Change to the BlindspotAssistance folder:

cd OpenVino-Blindspot-Assistance/BlindspotAssistance/

4. Download the necessary model

bash scripts/download_models.sh

5. Create a directory to build the tutorial in and change to it.

mkdir build
cd build

6. Before running each of the following sections, be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models:

source ../scripts/setupenv.sh

7. Compile

cmake -DCMAKE_BUILD_TYPE=Release ../
make

8. Move to the executable’s dir:

cd intel64/Release

4.2. Build with Docker

4.2.1. Prerequisites

• Docker. To install on Ubuntu, run:

sudo snap install docker

sudo groupadd docker

sudo usermod -aG docker $USER

4.2.2. Installation

1. Clone the repository at desired location:

git clone https://github.com/incluit/OpenVino-Blindspot-Assistance.git

2. Change to the top git repository:

cd OpenVino-Blindspot-Assistance

3. Build the docker:

make docker-build

4.3. Build in DockerHub

The master brach of this repository is built in Docker Hub:

https://hub.docker.com/r/openvinoincluit/blindspot-openvino

You can pull the blindspot assistance Docker image using:

docker pull openvinoincluit/blindspot-openvino

4.4. MQTT Broker Installation

We are using Mosquitto MQTT broker to handle all the messages from blindspot assistance.

To run Mosquitto broker in the base system use:

docker run -d --rm --name broker -p 1883:1883 eclipse-mosquitto

5. Usage

5.1. Run

Run the application with the -h option to see the usage message:

./blindspot-assistance -h

Options:

    -h                           Print a usage message.
    -m "<path>"                  Required. Path to an .xml file with a trained model.
      -l "<absolute_path>"       Required for CPU custom layers. Absolute path to a shared library with the kernel implementations.
          Or
      -c "<absolute_path>"       Required for GPU custom kernels. Absolute path to an .xml file with the kernel descriptions.
    -d "<device>"                Optional. Specify the target device for a network (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:<comma-separated_devices_list>" format to specify HETERO plugin. The demo looks for a suitable plugin for a specified device.
    -nc                          Optional. Maximum number of processed camera inputs (web cameras).
    -bs                          Optional. Batch size for processing (the number of frames processed per infer request).
    -nireq                       Optional. Number of infer requests.
    -n_iqs                       Optional. Frame queue size for input channels.
    -fps_sp                      Optional. FPS measurement sampling period between timepoints in msec.
    -n_sp                        Optional. Number of sampling periods.
    -pc                          Optional. Enable per-layer performance report.
    -t                           Optional. Probability threshold for detections.
    -no_show                     Optional. Do not show processed video.
    -no_show_d                   Optional. Optional. Do not show detected objects.
    -show_stats                  Optional. Enable statistics report.
    -duplicate_num               Optional. Enable and specify the number of channels additionally copied from real sources.
    -real_input_fps              Optional. Disable input frames caching for maximum throughput pipeline.
    -i                           Optional. Specify full path to input video files.
    -loop_video                  Optional. Enable playing video on a loop.
    -calibration                 Optional. Camera calibration.
    -show_calibration            Optional. Show camera calibration.
    -mqtt                        Optional. MQTT broker to handle all the messages.

5.1.1. Examples

1. Using 4 video files, using CPU and GPU:

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4

2. Using MQTT Broker:

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4 -mqtt

3. Using 3 video files and 1 camera (-nc 1):

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 -nc 1

4. Using 4 video files, showing statistics report (-show_stats):

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4 -show_stats

5. Using 4 video files, without video output (-no_show). Average performance is displayed on the console.

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4 -no_show

6. Using 4 video files, without detected objects (-no_show_d).

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4 -no_show_d

7. Using 4 video files, changing the probability threshold for detections (-t 0.1).

./blindspot-assistance -m ../../../models/FP32/pedestrian-and-vehicle-detector-adas-0001.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4 -t 0.1

5.1.2. Other models

You can also experiment by using different detection models, being the ones available up to now:

1. person-vehicle-bike-detection-crossroad-0078

2. person-vehicle-bike-detection-crossroad-1016

./blindspot-assistance -m ../../../models/FP32/person-vehicle-bike-detection-crossroad-0078.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4

./blindspot-assistance -m ../../../models/FP32/person-vehicle-bike-detection-crossroad-1016.xml -d HETERO:CPU,GPU -i ../../../data/BlindspotFront.mp4 ../../../data/BlindspotLeft.mp4 ../../../data/BlindspotRear.mp4 ../../../data/BlindspotRight.mp4

5.1.3. Run in Docker

1. Run the docker:

make docker-run

2. Run the example inside the Docker or try the detailed examples in Usage.

make run

3 If you get out of the Docker, you can run it again:

docker start blindspotcont
docker exec -it blindspotcont /bin/bash

6. Troubleshooting

1. If you receive the following message inside the Docker:

Gtk-WARNING **: 13:01:52.097: cannot open display: :0

Go outside the Docker container, run:

xhost +

Enter the Docker container and run it again.

7. [Optional] AWS (In Progress)

We also plan to send the data through MQTT using AWS IoT-Core. Using AWS may incur in a cost, so this will also be optional for you to run with/without it.