MVDNet

Robust Multimodal Vehicle Detection in Foggy Weather Using Complementary Lidar and Radar Signals, CVPR 2021.

Prerequisites

• Python 3.7
• Pytorch 1.5.0
• Detectron2
• Pycocotools

MVDNet uses an old version of Detectron2 (i.e., 0.1.1) with minor modifications. To download and install the compatible version:

git clone https://github.com/qiank10/detectron2.git
git checkout alt-0.1.1
cd detectron2 && pip install -e .

Install MVDNet

git clone https://github.com/qiank10/MVDNet.git
cd MVDNet && pip install -e .

Prepare Data

Download the Oxford Radar RobotCar Dataset. Currently, only the vehicles in the first data record (Date: 10/01/2019, Time: 11:46:21 GMT) are labeled. After unzipping the files, the directory should look like this:

# Oxford Radar RobotCar Data Record
|-- DATA_PATH
    |-- gt
    |-- radar
    |-- velodyne_left
    |-- velodyne_right
    |-- vo
    |-- radar.timestamps
    |-- velodyne_left.timestamps
    |-- velodyne_right.timestamps
    |-- ...

Prepare the radar data:

python data/sdk/prepare_radar_data.py --data_path DATA_PATH --image_size 320 --resolution 0.2

Prepare the lidar data:

python data/sdk/prepare_lidar_data.py --data_path DATA_PATH

Prepare the foggy lidar test set with specified fog density, e.g., 0.05:

python data/sdk/prepare_fog_data.py --data_path DATA_PATH --beta 0.05

The processed data is organized as follows:

# Oxford Radar RobotCar Data Record
|-- DATA_PATH
    |-- processed
        |-- radar
            |-- 1547120789640420.jpg
            |-- ...
        |-- radar_history
            |-- 1547120789640420_k.jpg   # The k-th radar frame preceding the frame at the timestamp 1547120789640420, k=1,2,3,4.
            |-- ...
        |-- lidar
            |-- 1547120789640420.bin
            |-- ...
        |-- lidar_history
            |-- 1547120789640420_k.bin   # Link to the k-th lidar frame preceding the frame at the timestamp 1547120789640420, k=1,2,3,4.
            |-- 1547120789640420_k_T.bin # Transform matrix between the k-th preceding lidar frame and the current frame.
            |-- ...
        |-- lidar_fog_0.05               # Foggy lidar data with fog density as 0.05
            |-- 1547120789640420.bin
            |-- ...
        |-- lidar_history_fog_0.05
            |-- 1547120789640420_k.bin
            |-- 1547120789640420_k_T.bin
            |-- ...

Both 2D and 3D labels are in

./data/RobotCar/object/

Train MVDNet

python ./tools/train.py --config ./configs/train_config.yaml

Evaluate MVDNet

python ./tools/eval.py --config ./configs/eval_config.yaml