Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion

arXiv 2026.05

Yiran Qiao, Yiren Lu, Yunlai Zhou, Disheng Liu, Linlin Hou, Rui Yang, Yu Yin, Jing Ma
Case Western Reserve University
Content image

Content Image

Style image

Style Image

3D Asset

Style-controllable 3D generation breaks down as soon as the target style drifts out of the generator’s training distribution. We find the bottleneck is not model capacity — structured 3D latents are already expressive, merely underused. We therefore introduce DiLAST: 2D Diffusion-based Latent Awakening for 3D Style Transfer, which turns a pretrained 2D diffusion model into a teacher: by aligning rendered views with the target style under diffusion guidance, it optimizes the structured 3D latents directly. A content image and an arbitrary style image are enough to produce a stylized 3D asset, and the method is plug-and-play across 3D generation backbones.

Abstract

3D asset generation plays a pivotal role in fields such as gaming and virtual reality, enabling the rapid synthesis of high-fidelity 3D objects from a single or multiple images. Building on this capability, enabling style-controllable generation naturally emerges as an important and desirable direction. However, existing approaches typically rely on style images that lie within or are similar to the training distribution of 3D generation models. When presented with out-of-distribution (OOD) styles, their performance degrades significantly or even fails. To address this limitation, we introduce DiLAST: 2D Diffusion-based Latent Awakening for 3D Style Transfer. Specifically, we leverage a pretrained 2D diffusion model as a teacher to provide rich and generalizable style priors. By aligning rendered views with the target style under diffusion-based guidance, our method optimizes the structured 3D latent representations for stylization. We observe that this limitation stems not from insufficient model capacity, but from the underutilization of structured 3D latents, which are inherently expressive. Despite being trained on comparatively limited data, 3D generation models can leverage 2D diffusion guidance to steer denoising toward specific directions in latent space, thereby producing diverse, OOD styles. Extensive experiments across diverse data and multiple 3D generation backbones demonstrate the effectiveness and plug-and-play nature of our approach.

Motivation

Comparison of the previous 3D style transfer pipeline and the DiLAST pipeline. Panel (a) shows the previous pipeline, where a content image and an out-of-distribution style image are fed through frozen flow transformers whose internal query, key and value attention is shared, and the resulting style-transferred 3D asset fails. Panel (b) shows the DiLAST pipeline, where a single content image produces a 3D structure latent that is decoded into a 3D asset in 3DGS format, rendered, and scored by a frozen 2D diffusion teacher whose attention distillation gradient flows back into the latent, succeeding on the same style.

Comparison of existing 3D style transfer methods and our DiLAST pipeline. (a) Existing methods rely solely on internal attention of 3D generative models and often fail when handling OOD styles. (b) In contrast, DiLAST leverages a pretrained 2D diffusion teacher, whose attention distillation gradients guide the denoising trajectory of 3D latents, successfully transferring arbitrary OOD styles.

Detailed Pipeline

Detailed overview of DiLAST. A content image and cube noise pass through a frozen flow transformer and decoder to a sparse structure, then noise is added and a second frozen flow transformer produces the 3D latent. A decoder feeds three parallel branches evaluated against a frozen 2D latent diffusion model: a content branch rendering RGB and opacity from the 3DGS asset, an optimization branch whose 3DGS asset is marked by a green bracket, and a style branch taking the style image. The resulting losses are backpropagated as a gradient on the 3D latent.

Overview of our DiLAST. We leverage a 2D LDM as a teacher and optimize the 3D latent through three parallel branches, guiding the generated 3D asset toward the target style. The 3D asset highlighted by the green bracket, generated at the final optimization step, is the final output.

More Results

Each column is one example. From the content image in the first row and the style image in the second, DiLAST produces the stylized 3D asset shown in the third row.

Content
Content image for example 1
Content image for example 2
Content image for example 3
Content image for example 4
Content image for example 5
Content image for example 6
Style
Style image for example 1
Style image for example 2
Style image for example 3
Style image for example 4
Style image for example 5
Style image for example 6
Video

Baseline Comparison

Qualitative comparison against 3D style transfer baselines. Each column is one style image (top row) and each row is one content image (left of the row), repeated for the three methods. MorphAny3D and StyleSculptor drift away from the content geometry once the style is out of distribution, while DiLAST keeps the object intact and still takes on the style. Every result is a rendered orbit of the generated 3D asset.

StyleContent
Style image 66
Style image 72
Style image 84
Style image 112
Style image 119
MorphAny3D
Content image, rock monster
Content image, lantern
StyleSculptor
Content image, rock monster
Content image, lantern
DiLAST (Ours)
Content image, rock monster
Content image, lantern

Why Not Stylize in 2D First?

A natural alternative is to run 2D style transfer on the content image first and then lift the stylized image to 3D. That route runs into a trade-off with no good setting. When content preservation is too strong, the style cannot be effectively transferred. Conversely, when content preservation is too weak, the stylized image deviates further from the training distribution of the 3D generative model, leading to poor generation quality. In practice, it is often difficult for 2D style transfer to find a suitable level of content preservation that remains “recognizable” to the 3D model. In contrast, DiLAST directly guides the 3D latents, indirectly injecting an appropriate degree of style into the latent space while achieving suitable content preservation, without being affected by the 2D stylized image.

Input
2D Stylization → 3D Generation  ✗
DiLAST (Ours)  ✓
Content
Style
Stylized Imagehigher content preserving
→ 3D Asset
Stylized Imagelower content preserving
→ 3D Asset
3D Assetfrom the 3D latents
Content image, dwarf
Style image, cloud painting
2D stylized dwarf in the cloud painting style, higher content preserving
2D stylized dwarf in the cloud painting style, lower content preserving
Content image, dwarf
Style image, anime line art
2D stylized dwarf in the anime line art style, higher content preserving
2D stylized dwarf in the anime line art style, lower content preserving

Citation

If you find our work helpful, please consider citing us:

@article{qiao2026structured,
  title={Structured {3D} Latents Are Surprisingly Powerful: Unleashing Generalizable Style with {2D} Diffusion},
  author={Qiao, Yiran and Lu, Yiren and Zhou, Yunlai and Liu, Disheng and Hou, Linlin and Yang, Rui and Yin, Yu and Ma, Jing},
  journal={arXiv preprint arXiv:2605.04412},
  year={2026}
}