Tasks

ScanNet Benchmark Tasks

Train/val/test split for ScanNet is given in Benchmark. Label mappings (scannet-labels.combined.tsv) can be downloaded with the ScanNet task data release. Download with along with the task data (--task_data) or by itself (--label_map). ScanNet label ids listed under id.

Trained models can be downloaded with the ScanNet data release.

Installation

Training tasks use Torch7, with torch packages cudnn, cunn, hdf5, xlua.

Train/test data from the ScanNet release should be placed into a data folder in the torch folder of the respective task, e.g., for 3D Object Classification, unzip the obj_classification.zip into Tasks/ObjectClassification/torch/data. (Or edit the paths in the [phase]_shape_voxel_data_list.txt and input the train/test paths to train.lua.)

Code to generate the training data from ScanNet data to come soon.

Tasks

3D Object Classification

Classification of partially-scanned objects, voxelized to 30x30x30. We use the 3D CNN framework of Qi et. al. with an additional data channel encoding known/unknown voxels according to the camera scanning trajectory. The object classes used are given in classes_ObjClassification-ShapeNetCore55.txt, using ShapeNetCore55 label ids.

• Training:
  run

th train_partial.lua --train_data [path to train h5 file list] --test_data [path to test h5 file list]

with the appropriate paths to the train/test data (use '--help' to see more options)

• Citation:

@article{qi2016volumetric,
    title={Volumetric and Multi-View CNNs for Object Classification on 3D Data},
    author={Charles Ruizhongtai Qi and Hao Su and Matthias Nie{\ss}ner and Angela Dai and Mengyuan Yan and Leonidas Guibas},
    journal={Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
    year={2016}
}

3D Object Retrieval

Nearest neighbor 3d model retrieval based on the features of the 3D object classification model. The code in ObjectRetrieval uses the feature extraction and nearest neighbor code from 3dmodel_feature.

To compute nearest neighbors, first extract the features from both the query and the database by:

th generate_feat.lua --model [model] --h5_list_path [path to h5 file list to generate features for] --partial_data [use this flag if partial data] --file_label_file [assocation from h5 data to names and labels] --output_file [output file (features)] --output_name_file [output file (names)] --classes_file [optional, defines set of classes to consider]

then compute the nearest neighbors with:

python nearest_neighbor.py --query_feats [feature file from generate_feat] --database_feats [feature file from generate_feat] --database_models [names corresponding to database feats] --output [output file] 

Semantic Voxel Labeling

Prediction of class labels per voxel for voxels on the surface of a scan. Voxels of a scan are labeled column by column using the surrounding neighborhood information (31x31x62 neighborhood for labeling the 1x1x62 center column). The object classes used for ScanNet are given in classes_SemVoxLabel-nyu40id.txt, using nyu40 label ids.

• Training:

th train.lua --train_data [path to train h5 file list] --test_data [path to test h5 file list]

with the appropriate paths to the train/test data (use '--help' to see more options)