AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

Muyao Niu1,2,   Mingdeng Cao1,   Yifan Zhan1,2,   Qingtian Zhu1,   Mingze Ma1,   Jiancheng Zhao1,  
Yanhong Zeng2,   Zhihang Zhong2,*,   Xiao Sun2,*,   Yinqiang Zheng1

1 The University of Tokyo

,

2 Shanghai AI Lab

,

* Corresponding Author

arXiv 2025

Paper Code 🤗 HuggingFace Model

TL;DR: We control the Video Diffusion Models with 3DGS Avatars to insert and animate anyone into any scene following given motion sequence.

Character Image

Avatar-Background Video

Output Video

Overview & Architecture

We introduce AniCrafter, a diffusion-based human-centric animation model that can seamlessly integrate and animate a given character into open-domain dynamic backgrounds while following given human motion sequences. Built on cutting-edge Image-to-Video (I2V) diffusion architectures, our model incorporates an innovative "avatar-background" conditioning mechanism that reframes opendomain human-centric animation as a restoration task, enabling more stable and versatile animation outputs.

Instead of predicting the target video, our model predicts “reference image + target video” based on the pretrained Image-to-Video (I2V) diffusion transformer model. 3D Convolution layers and mask embeddings are leveraged to formulate useful spatial-temporal features from both the avatar-background video and the SMPL-X mesh video.

Cross Character-Scene Animation

Avatar-Background

Character Image + Output

Avatar-Background

Character Image + Output

Self-Reenactment Animation

Character

Avatar-Bkgd

Output Video

Ground-Truth

Character

Avatar-Bkgd

Output Video

Ground-Truth

Character

Avatar-Bkgd

Output Video

Ground-Truth

BibTeX

@article{niu2025anicrafter,
        title={AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models},
        author={Niu, Muyao and Cao, Mingdeng and Zhan, Yifan and Zhu, Qingtian and Ma, Mingze and Zhao, Jiancheng and Zeng, Yanhong and Zhong, Zhihang and Sun, Xiao and Zheng, Yinqiang},
        journal={arXiv preprint arXiv:2505.20255},
        year={2025}
        }