Neural Scene Flow Prior

Xueqian Li, Jhony Kaesemodel Pontes, Simon Lucey

Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS, 2021), Spotlight presentation.

Project page: https://lilac-lee.github.io/Neural_Scene_Flow_Prior/.

arXiv link: https://arxiv.org/pdf/2111.01253.pdf.

Update:

The code for preprocessing the dataset to get the pseudo scene flow is released! You can find it in preprocess_sf_argoverse.py and preprocess_sf_nuscenes.py.

---

• Scene flow results on Argoverse

  

• Point cloud integration (left: densified point cloud; right: sparse point cloud)

  

---

Prerequisites

This code is based on PyTorch implementation, and tested on torch=1.6.0 with CUDA 10.1 OR torch=1.7.1 with CUDA 10.2.

For a detailed installation guide, please go to requirements.txt.

Dataset

We provide four datasets we used in our paper. You may download datasets used in the paper from these anonymous links:

• KITTI (266MB)

• Argoverse (370MB)

• nuScenes (73MB)

• FlyingThings3D (948MB)

After you download the dataset, you can create a symbolic link in the ./dataset folder as ./dataset/kitti, ./dataset/argoverse, ./dataset/nuscenes, and ./dataset/flyingthings.

---

Optimization

Since we use neural scene flow prior for runtime optimization, our method does not include any "training".

Just run following lines for a simple optimization on a small KITTI Scene Flow dataset (only 50 testing samples)

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path /scratch/ag7644/kitti \
--exp_name KITTI_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 70 \
--visualize

You can then play with these configurations. We provide commands we used to generate results in the small point coud (2048 points) experiments and large point cloud (all points included) experiments.

1. small point cloud (2048 points)

Waymo Scene Flow

python optimization.py \
--dataset WaymoSceneFlowDataset \
--dataset_path /scratch/ag7644/waymo_sf_debug11 \
--exp_name waymo_sf_debug11_2_vis \
--batch_size 1 \
--use_all_points \
--iters 500 \
--compute_metrics \
--hidden_units 128 \
--lr 0.001  \
--backward_flow  \
--early_patience 300 \
--visualize \
--load_model_path "/scratch/ag7644/nsfp/checkpoints/waymo_sf_train_1/model/model_best.pth"

--use_all_points \

KITTI Scene Flow

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path /scratch/ag7644/kitti \
--exp_name KITTI_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 70 \
--visualize

Argoverse Scene Flow

python optimization.py \
--dataset ArgoverseSceneFlowDataset \
--dataset_path /scratch/ag7644/argoverse \
--exp_name Argoverse_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

nuScenes Scene Flow

python optimization.py \
--dataset NuScenesSceneFlowDataset \
--dataset_path dataset/nuscenes \
--exp_name Argoverse_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

FlyingThings3D

python optimization.py \
--dataset FlyingThings3D \
--dataset_path dataset/flyingthings \
--exp_name FlyingThings_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

2. dense point cloud (all points included)

Waymo Scene Flow

python optimization.py \
--dataset WaymoSceneFlowDataset \
--dataset_path /scratch/ag7644/waymo_sf/train \
--exp_name waymo_sf_train_11 \
--batch_size 1 \
--iters 5000 \
--use_all_points \
--compute_metrics \
--hidden_units 128 \
--lr 0.001  \
--backward_flow  \
--early_patience 300

KITTI Scene Flow

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path /scratch/ag7644/kitti \
--exp_name KITTI \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--use_all_points \
--hidden_units 128 \
--lr 0.001 \
--early_patience 100 \
--visualize

Argoverse Scene Flow

python optimization.py \
--dataset ArgoverseSceneFlowDataset \
--dataset_path /scratch/ag7644/argoverse/Argoverse_SceneFlow_remove_ground \
--partition train \
--exp_name Argoverse_train4 \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--use_all_points \
--hidden_units 128 \
--lr 0.003 \
--backward_flow \
--early_patience 100 \
--visualize \
--load_model_path "/scratch/ag7644/nsfp/checkpoints/Argoverse_train2/model/model_best.pth"

python optimization.py \
--dataset ArgoverseSceneFlowDataset \
--dataset_path /scratch/ag7644/argoverse/Argoverse_SceneFlow_ins \
--partition train \
--exp_name Argoverse_train_ins_bbox_4 \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--use_all_points \
--hidden_units 128 \
--lr 0.003 \
--backward_flow \
--early_patience 100 \
--visualize \
--load_model_path "/scratch/ag7644/nsfp/checkpoints/Argoverse_train_ins_3/model/model_latest.pth"

---

Contributing

If you find the project useful for your research, you may cite,

@article{li2021neural,
  title={Neural Scene Flow Prior},
  author={Li, Xueqian and Kaesemodel Pontes, Jhony and Lucey, Simon},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}