Question about reproducing results using provided config file.

[linleyzoe]

Hi, thanks for your inspiring work!
Following vic3d/late-fusion-pointcloud/pointpillars/README.md, the reproduced results using trainval_config_v.py and trainval_config_v.py are not as good as those using your provided checkpoints (inf-model & veh-model).

the reproduced results

car 3d IoU threshold 0.30, Average Precision = 60.87​
car 3d IoU threshold 0.50, Average Precision = 49.24​
car 3d IoU threshold 0.70, Average Precision = 29.07​
car bev IoU threshold 0.30, Average Precision = 64.03​
car bev IoU threshold 0.50, Average Precision = 54.57​
car bev IoU threshold 0.70, Average Precision = 44.08​
Average Communication Cost = 927.07 Bytes

results from provided checkpoints

car 3d IoU threshold 0.30, Average Precision = 63.40
car 3d IoU threshold 0.50, Average Precision = 53.36
car 3d IoU threshold 0.70, Average Precision = 37.28
car bev IoU threshold 0.30, Average Precision = 65.26
car bev IoU threshold 0.50, Average Precision = 59.16
car bev IoU threshold 0.70, Average Precision = 50.53
Average Communication Cost = 897.99 Bytes

How to reproduce the results from your provided checkpoints?
Any advice would be greatly appreciated!

[hhjj233]

In fact, I think the writer didn't train the network carefully.
I choose VIC-Async-2 as the dataset, in which infrastructure has 2 frames delay than vehicle-side for each pair of pointcloud data.
And my fusion result is much better than the result reported in the paper.
Here is my result

car 3d IoU threshold 0.30, Average Precision = 59.46
car 3d IoU threshold 0.50, Average Precision = 48.30
car 3d IoU threshold 0.70, Average Precision = 28.85
car bev IoU threshold 0.30, Average Precision = 62.70
car bev IoU threshold 0.50, Average Precision = 53.49
car bev IoU threshold 0.70, Average Precision = 42.79
Average Communication Cost = 879.31 Bytes

[hhjj233]

And I wonder which setting did you choose in the experiment, in which inf-model and veh-model provided by author, the result is not the same as any results listed in the paper.

[haibao-yu]

Yes, all experiment results of point cloud is much better than the paper report, because we keep improving the experiment details and code.

[haibao-yu]

Hi, thanks for your inspiring work! Following vic3d/late-fusion-pointcloud/pointpillars/README.md, the reproduced results using trainval_config_v.py and trainval_config_v.py are not as good as those using your provided checkpoints (inf-model & veh-model).

the reproduced results

car 3d IoU threshold 0.30, Average Precision = 60.87​
car 3d IoU threshold 0.50, Average Precision = 49.24​
car 3d IoU threshold 0.70, Average Precision = 29.07​
car bev IoU threshold 0.30, Average Precision = 64.03​
car bev IoU threshold 0.50, Average Precision = 54.57​
car bev IoU threshold 0.70, Average Precision = 44.08​
Average Communication Cost = 927.07 Bytes

results from provided checkpoints

car 3d IoU threshold 0.30, Average Precision = 63.40
car 3d IoU threshold 0.50, Average Precision = 53.36
car 3d IoU threshold 0.70, Average Precision = 37.28
car bev IoU threshold 0.30, Average Precision = 65.26
car bev IoU threshold 0.50, Average Precision = 59.16
car bev IoU threshold 0.70, Average Precision = 50.53
Average Communication Cost = 897.99 Bytes

How to reproduce the results from your provided checkpoints? Any advice would be greatly appreciated!

Did you prepare the data as the instructions?