%0 Conference Proceedings
%T 3D-Aware Multi-Class Image-to-Image Translation with NeRFs
%A Senmao Li
%A Joost Van de Weijer
%A Yaxing Wang
%A Fahad Shahbaz Khan
%A Meiqin Liu
%A Jian Yang
%B 36th IEEE Conference on Computer Vision and Pattern Recognition
%D 2023
%F Senmao Li2023
%O LAMP
%O exported from refbase (http://158.109.8.37/show.php?record=3920), last updated on Mon, 22 Jan 2024 11:23:35 +0100
%X Recent advances in 3D-aware generative models (3D-aware GANs) combined with Neural Radiance Fields (NeRF) have achieved impressive results. However no prior works investigate 3D-aware GANs for 3D consistent multiclass image-to-image (3D-aware 121) translation. Naively using 2D-121 translation methods suffers from unrealistic shape/identity change. To perform 3D-aware multiclass 121 translation, we decouple this learning process into a multiclass 3D-aware GAN step and a 3D-aware 121 translation step. In the first step, we propose two novel techniques: a new conditional architecture and an effective training strategy. In the second step, based on the well-trained multiclass 3D-aware GAN architecture, that preserves view-consistency, we construct a 3D-aware 121 translation system. To further reduce the view-consistency problems, we propose several new techniques, including a U-net-like adaptor network design, a hierarchical representation constrain and a relative regularization loss. In exten-sive experiments on two datasets, quantitative and qualitative results demonstrate that we successfully perform 3D-aware 121 translation with multi-view consistency. Code is available in 3DI2I.
%U https://www.computer.org/csdl/proceedings-article/cvpr/2023/012900m2652/1POUWa130nS
%U http://dx.doi.org/10.1109/CVPR52729.2023.01217
%P 12652-12662