🎬 Populate-A-Scene:
Affordance-Aware Human Video Generation

Mengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang

🏫 Univeristy of Washington, 🏢 Meta GenAI

📄 Preprint 2025 · ICCV Reviews: 4/5/6


Paper Supplement Video Gallery

We add moving humans to scenes, without knowing their pose or position. 🕺💃
Check out our video gallery for more results! 📷

Cond 1 Cond 2
Cond 3 Cond 4
Cond 5 Cond 6
Cond 7 Cond 8

Abstract

Can a video generation model be repurposed as an interactive world simulator? We explore the affordance perception potential of text-to-video models by teaching them to predict human-environment interaction. Given a scene image and a prompt describing human actions, we fine-tune the model to insert a person into the scene, while ensuring coherent behavior, appearance, harmonization, and scene affordance. Unlike prior work, we infer human affordance for video generation (i.e., where to insert a person and how they should behave) from a single scene image, without explicit conditions like bounding boxes or body poses. An in-depth study of cross-attention heatmaps demonstrates that we can uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets.

System Description

Pipeline diagram

We start by removing humans from raw frames to create synthetic empty-scene and human-video data pairs. We employ a dual-conditioning mechanism, using channel concatenation and cross-attention, to condition the T2V model on an additional scene image. We design a fusion module to facilitate interactions between image and action-text features while locating the desired action position. The fine-tuning pipeline trains a Transformer architecture based on flow matching framework.

Data: Training and Evaluation Benchmark

Pipeline diagram 1
Pipeline diagram 2

Left: Training dataset. Top row shows single-person data, while the bottom row shows double-person data. Within each row, the top figure presents the raw first frame of the video, while the bottom figure(s) show the result after detecting and removing humans from the scene.
Right: Synthetic action descriptions evaluation benchmark. We use a vision language AI agent to decide palusible actions in a scene, and rewrite the action into prompts.

Two-Person Video Results
















Multi-Prompt Video Results














BibTeX


          @misc{shan2025populate,
            title={Populate-A-Scene: Affordance-Aware Human Video Generation},
            author={Mengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang},
            year={2025},
            archivePrefix={arXiv},
            primaryClass={cs.CV}
          }