Unofficial implement of Live3DPortrait

This is an unofficial implement of Live3DPortrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis in Pytorch.

output.mp4

Pretrained Model

The checkpoints can be downloaded from here.

Place them under ./pretrained_models/ folder.

└── root
    ...
    └── pretrained_models
        └── encoder_render.pt
        └── ffhqrebalanced512-128.pth
        └── model_ir_se50.pth        

Usage

python scripts/inference.py --input ./imgs/input.png --output ./imgs/output.png --checkpoint ./checkpoints/ckpt.pth 

Notice that the input image should be preprocessed folowing the EG3D. Note that the LT model implement has some difference with the original implement, but achieve the nearly same performance.

TODO

• Upload inference code and checkpoint.
• Release Training code.
• Release LT model and checkpoint.

Acknowledgement

This repo is based on triplanenet, EG3D. Thanks to their great work!

@inproceedings{trevithick2023,
  author = {Alex Trevithick and Matthew Chan and Michael Stengel and Eric R. Chan and Chao Liu and Zhiding Yu and Sameh Khamis and Manmohan Chandraker and Ravi Ramamoorthi and Koki Nagano},
  title = {Real-Time Radiance Fields for Single-Image Portrait View Synthesis},
  booktitle = {ACM Transactions on Graphics (SIGGRAPH)},
  year = {2023}
}

@inproceedings{Chan2022,
  author = {Eric R. Chan and Connor Z. Lin and Matthew A. Chan and Koki Nagano and Boxiao Pan and Shalini De Mello and Orazio Gallo and Leonidas Guibas and Jonathan Tremblay and Sameh Khamis and Tero Karras and Gordon Wetzstein},
  title = {Efficient Geometry-aware {3D} Generative Adversarial Networks},
  booktitle = {CVPR},
  year = {2022}
}

@article{bhattarai2024triplanenet,
  title={TriPlaneNet: An Encoder for EG3D Inversion},
  author={Bhattarai, Ananta R. and Nie{\ss}ner, Matthias and Sevastopolsky, Artem},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2024}
}