Hi, I'm Muyao Niu. I am a 2nd year Ph.D. student in the Department of Mechano-Informatics, the University of Tokyo (UTokyo). My supervisor is Prof. Yinqiang Zheng.
I received my master's degree from The University of Tokyo in 2024, and B.E. degree from Dalian University of Technology in 2022.
My research interests include:
Generative Modeling (video & 3D): I explore video and 3D representations for controllable generation. I believe that hybrid representations are the future of generative modeling.
Works: MOFA-Video, WorldSculpt, AniCrafter, RASC, CompoSIA, CV-VAE, StereoCrafter, Surprise-Forcing, etc.
NeRF & 3DGS: I study 3D reconstruction from various observations by jointly modeling scene structure and imaging process.
Works: MAD-Avatar, 3DarkFusion, RS-NeRF, Visionary, ToMiE, etc.
Computational Photography: I believe that photography-aware modeling will be even more important when AGI comes.
Works: VCSD, AdvNIR, NVEU, SpaceDG, Event-Attack, etc.
ECCV, 2024
project page / paper / code
We introduce MOFA-Video to adapt motions from different domains to the frozen Video Diffusion Model. MOFA-Video can effectively animate a single image using various types of control signals, including trajectories, keypoint sequences, and their combinations.
Technical Report, 2026
project page / paper / code
WorldSculpt produces a compositional mesh representation for very complex scenes consisting of hundreds of individual objects given grounded RGB videos.
ACM MM, 2026
project page / paper / code
We leverage "3DGS Avatar + Background Video" as hybrid guidance for the video diffusion model to insert and animate anyone into any scene following given motion sequence.
CVPR, 2023
We designed an optimal illumination spectrum in the VIS-NIR range by considering human vision constraints, which significantly improves translation performance. A fully differentiable model was proposed, which includes the imaging process, human visual perception, and the enhancement network.
ACM MM, 2023
We introduced an innovative approach that passively manipulates the intensity distribution of NIR images and developed a 3D-aware, black-box attack algorithm to target deep learning-based NIR-powered human detection systems.
ICCV, 2023
We addressed the issue of collecting data for utilizing NIR images to improve low-light VIS videos. Physics-inspired algorithms are designed to simulate pseudo paired data of NIR and VIS images, simulating day-to-night situations. We then trained an enhancement network using the generated pseudo data.
Technical Report, 2026
project page / paper / code
ACM MM, 2026
project page / paper / code
ECCV, 2026
Surprise Forcing is a training-free framework that uses "surprise" to keep the most informative frames discarded by the sliding-window KV cache (via a surprise-aware memory bank) and to adaptively cut denoising steps for easy chunks—preserving long-range scene consistency and narrative coherence in autoregressive diffusion long-video generation.
arXiv, 2026
project page / paper / code
We introduce SpaceDG, the first large-scale dataset for degradation-aware spatial intelligence, and SpaceDG-Bench, a human-verified benchmark for evaluating MLLMs under visual degradations.
ECCV, 2026
project page / paper / code
CompoSIA is a powerful simulator for synthesizing rare driving scenes, enabling explicit control over structure, identity, and ego action.
arXiv, 2025
project page / paper / code
We build an open platform via WebGPU and ONNX Runtime, enabling real-time rendering of diverse Gaussian Splatting variants (3DGS, MLP-based 3DGS, 4DGS, Neural Avatars).
arXiv, 2025
We present R3-Avatar, incorporating a temporal codebook, to overcome the inability of human avatars to be both animatable and of high-fidelity rendering quality.
ICCV, 2025
We propose Tree-NeRV, a novel tree-structured feature representation for efficient and adaptive video encoding. Unlike conventional approaches, Tree-NeRV organizes feature representations within a Binary Search Tree (BST), enabling non-uniform sampling along the temporal axis.
ICCV, 2025
We highlight a critical yet often overlooked factor in most 3D human tasks, namely modeling complicated 3D human with hand-held objects or loose-fitting clothing by modeling the exoskeleton of the human body.
AAAI, 2025
We developed an end-to-end adversarial framework for event-driven pedestrian detection, framing the design of adversarial clothing textures as a 2D texture optimization problem.
Technical Report, 2024
We present a novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience. Leveraging foundation models as priors, our approach boosts the performance to ensure the high-fidelity generation required by the display devices.
ECCV, 2024
project page / paper / code
NeurIPS, 2024
project page / paper / code
We propose CV-VAE that is compatible with existing image and video models trained with SD image VAE. Our video VAE provides a truly spatio-temporally compressed latent space for latent generative video models, as opposed to uniform frame sampling.
TIP, 2023
We proposed the Region-Assisted Sketch Colorization (RASC) method, which uses a 'Region Map' to better utilize regional information within the sketch, enhancing the perception of region-wise features.
Pattern Recognition, 2023
We propose a multi-scale Transformation Region Enhancement Network (TRE-Net) to enhance the learning on geometric transformation regions.
CVPR 2025 Outstanding Reviewer
BOOST NAIS Special Research Scholarship (3,900,000 JPY / year), The University of Tokyo, 2024.10 - 2027.09
WING-CFS Special Research Assistant Scholarship (2,160,000 JPY / year), The University of Tokyo, 2023.04 - 2024.09
Reviewer: CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, ACM MM, BMVC, TMLR.