Advances in 3D Neural Stylization: A Survey

TL;DR: A review of 3D neural stylization papers, mainly neural stylization on 3D data with image or text reference.

The survey delves into the realm of neural stylization on diverse 3D representations, including meshes, point clouds, volume, and neural fields. The neural stylization with visual, textual and geometric features retrieved from large-scale neural models empowers artistic, photorealistic, and semantic style transformation of the geometry and appearance of 3D scenes.

Abstract

Modern artificial intelligence offers a novel and transformative approach to creating digital art across diverse styles and modalities like images, videos and 3D data, unleashing the power of creativity and revolutionizing the way that we perceive and interact with visual content. This paper reports on recent advances in stylized 3D asset creation and manipulation with the expressive power of neural networks. We establish a taxonomy for neural stylization, considering crucial design choices such as scene representation, guidance data, optimization strategies, and output styles. Building on such taxonomy, our survey first revisits the background of neural stylization on 2D images, and then presents in-depth discussions on recent neural stylization methods for 3D data, accompanied by a benchmark evaluating selected mesh and neural field stylization methods. Based on the insights gained from the survey, we highlight the practical significance, open challenges, future research, and potential impacts of neural stylization, which facilitates researchers and practitioners to navigate the rapidly evolving landscape of 3D content creation using modern artificial intelligence.

Taxonomy

Taxonomy of neural stylization.

The taxonomy for neural stylization methods consists of the following aspects:

Representations. We categorize stylization methods based on data representations such as image, mesh, volume, point cloud, and neural field.
Neural Style Feature. We categorize based on image visual features, textual semantic features, or 3D latent features derived from pre-trained models, typically neural classifiers or generative models.
Optimization. This refers to optimization-based or prediction-based stylization methods with single, multiple, or arbitrary styles supported.
Stylization Genres. This refers to different types of stylization, mainly including geometry stylization operating on asset shape and surface patterns, and appearance stylization focusing on color, texture and visual patterns to align with specific styles from artistic paintings to realistic concepts.

Application

The burgeoning technologies for generating and manipulating 3D assets are unleashing the power of creativity and revolutionizing the way that we perceive and interact with visual content. 3D neural stylization sheds light on a new paradigm of providing infinite aesthetic possibilities from classic paintings to futuristic concepts, enhanced immersive experiences for virtual and augmented reality environments, seamless integration for cross-industry applications including advertising and marketing, fashion and product design, film and game development, architecture and environment visualization, interactive education and learning, etc.

Paper Collection

3D Representation

Guidance

Optimization

Style Genre

Citation


@article{chen2023advances3dstyle,
    title={Advances in 3D Neural Stylization: A Survey},
    author={Chen, Yingshu and Shao, Guocheng and Shum, Ka Chun and Hua, Binh-Son and Yeung, Sai-Kit},
    journal={arXiv preprint arXiv:2311.18328},
    year={2023}
}

Acknowledgement

The paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. HKUST 16202323) and an internal grant from HKUST (R9429).
Binh-Son Hua is supported by Research Ireland under the Research Ireland Frontiers for the Future Programme - Project, award number 22/FFP-P/11522.

This webpage is maintained by Chen Yingshu and Shao Guocheng.