Welcome to Hulk! Hulk is a multimodel human-centric generalist model, capable of addressing 2D vision, 3D vision, skeleton-based, and vision-language human-centric tasks. Unlike many existing human-centric foundation models that did not explore 3D and vision-language tasks for human-centric and required task-specific finetuning, Hulk condensed various task-specific heads into two general heads, one for discrete representations, e.g., languages, and the other for continuous representations, e.g., location coordinates. Unifying these tasks enables Hulk to treat diverse human-centric tasks as modality translation, integrating knowledge across a wide range of tasks. For more details, please take a look at our paper Hulk: A Universal Knowledge Translator for Human-centric Tasks.

• Dec. 2024 The training and testing datasets are released.
• Apr. 2024 A pretrained Hulk is released on 🤗 Hugging Face Models!
• Apr. 2024 Project page with demos is released at Hulk.
• Mar. 2024 Training and inference code are released!
• Dec. 2023 Hulk is released on ArXiv!

This codebase has been developed with python version 3.9, pytorch 2.0.0, cuda 11.8 and torchvision 0.15.0. We recommend using the same version to avoid potential issues.

pip install -r requirements.txt

Also, download bert-base-uncased from huggingface and put it under experiments/release/.

Please refer to the datasets for more details.

Download pre-trained MAE weights from here and put it under core/models/backbones/pretrain_weights/.

We use 10 nodes (80 A100 GPUs) for training with the following command:

cd experiments/release
sh train.sh 80 Hulk_vit-B

A pretrained Hulk will be soon available at 🤗 Hugging Face Models. Download it, put it under the folder experiments/release/checkpoints/Hulk_vit-B (first mkdir -p experiments/release/checkpoints/Hulk_vit-B), then use the following command to evaluate the model on the test set.

cd experiments/release
sh batch_eval.sh 1 Hulk_vit-B 

We use the plain ViT as our backbone, develop four modality-specific tokenizers and de-tokenizers to cover 2D vision, 3D vision, skeleton-based, and vision-language human-centric tasks. Hulk has achieved state-of-the-art results on various human-centric tasks.

♣: We find that the performance of image caption and monocular 3D human pose and mesh recovery is not as good as the direct evaluation, indicating that overfitting may occur during finetuning.

If you have any problem about our paper & code, feel free to contact Yizhou Wang and Yixuan Wu.

If you find this work useful, please consider citing:

@article{wang2023hulk,
  title={Hulk: A Universal Knowledge Translator for Human-Centric Tasks},
  author={Wang, Yizhou and Wu, Yixuan and Tang, Shixiang and He, Weizhen and Guo, Xun and Zhu, Feng and Bai, Lei and Zhao, Rui and Wu, Jian and He, Tong and others},
  journal={arXiv preprint arXiv:2312.01697},
  year={2023}
}