Animating Street View

University of Washington
SIGGRAPH Asia 2023

We make street scenes alive by inserting naturally behaving pedestrians and vehicles.


We present a system that automatically brings street view imagery to life by populating it with naturally behaving, animated pedestrians and vehicles. Our approach is to remove existing people and vehicles from the input image, insert moving objects with proper scale, angle, motion and appearance, plan paths and traffic behavior, as well as render the scene with plausible occlusion and shadowing effects. The system achieves these by reconstructing the still image street scene, simulating crowd behavior, and rendering with consistent lighting, visibility, occlusions, and shadows. We demonstrate results on a diverse range of street scenes including regular still images and panoramas.

System Description


Our system has three major components. In Stage 1, we reason about the scene by predicting its semantic segmentation labels, depth values, sun direction and intensity, as well as shadow regions. We additionally determine walking and driving regions for adding pedestrians and cars (red straight lines: lane detection; blue points: origin and destination points). In Stage 2, we simulate the pedestrians in a 2D bird's eye view representation (BEV) of the scene, and simulate car movements with predicted lanes (four colors correspond to four predicted path, both in BEV and scene images). If there is a detected crosswalk, we also simulate the traffic behavior by controlling a traffic light. In Stage 3, we render the scene with the estimated lighting, shadows, and occlusions. The whole pipeline is automated.

Another Carousel


Video Presentation



          title={Animating Street View},
          author={Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz},
          booktitle={Proceedings of ACM SIGGRAPG Asia 2023},
          doi = {},