RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark (ECCV 2024)

[paper] [dataset]

Abstract

Traditional methods for human localization and pose estimation (HPE), which mainly rely on RGB images as an input modality, confront substantial limitations in real-world applications due to privacy concerns. In contrast, radar-based HPE methods emerge as a promising alternative, characterized by distinctive attributes such as through-wall recognition and privacy-preserving, rendering the method more conducive to practical deployments. This paper presents a Radar Tensor-based human pose (RT-Pose) dataset and an open-source benchmarking framework. RT-Pose dataset comprises 4D radar tensors, LiDAR point clouds, and RGB images, and is collected for a total of 72k frames across 240 sequences with six different complexity level actions. The 4D radar tensor provides raw spatio-temporal information, differentiating it from other radar point cloud-based datasets. We develop an annotation process, which uses RGB images and LiDAR point clouds to accurately label 3D human skeletons. In addition, we propose HRRadarPose, the first single-stage architecture that extracts the high-resolution representation of 4D radar tensors in 3D space to aid human keypoint estimation. HRRadarPose outperforms previous radar-based HPE work on the RT-Pose benchmark. The overall HRRadarPose performance on the RT-Pose dataset, as reflected in a mean per joint position error (MPJPE) of 9.91cm, indicates the persistent challenges in achieving accurate HPE in complex real-world scenarios.

RT-Pose is available on Hugging Face.

Data Processing

Please check data processing

Baseline Method and Evaluations

Installation

We recommend using conda to manage the Python environment:

conda create -n rt_pose python=3.9 -y
conda activate rt_pose
pip install -r requirements-torch.txt --index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt
cd det3d/ops/dcn && python setup.py build_ext --inplace
cd ../../..

Training

To train the baseline method:

python ./tools/train.py configs/<config file>

For more details about the command options,

python ./tools/train.py -h

Evaluations

To evaluate the baseline method:

python ./tools/test.py <config file> --checkpoint <model checkpoint> --testset

For more details about the command options,

python ./tools/test.py -h

License

RT-Pose is released under CC BY-NC-SA.

Main Contributors

Yuan-Hao Ho ([email protected]), Jen-Hao(Andy) Cheng([email protected])

Citing RT-Pose

@article{rtpose2024,
      title={RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark}, 
      author={Yuan-Hao Ho and Jen-Hao Cheng and Sheng Yao Kuan and Zhongyu Jiang and Wenhao Chai and Hsiang-Wei Huang and Chih-Lung Lin and Jenq-Neng Hwang},
      journal={arXiv preprint arXiv:2407.13930},
      year={2024}
}

Acknowlegement

Our project could not have been accomplished without leveraging several exceptional open-source codebases.

• centerpoint
• det3d