Animating Street View

Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz
University of Washington
SIGGRAPH Asia 2023
Paper Supplementary Video arXiv

We make street scenes alive by inserting naturally behaving pedestrians and vehicles.

Abstract

We present a system that automatically brings street view imagery to life by populating it with naturally behaving, animated pedestrians and vehicles. Our approach is to remove existing people and vehicles from the input image, insert moving objects with proper scale, angle, motion and appearance, plan paths and traffic behavior, as well as render the scene with plausible occlusion and shadowing effects. The system achieves these by reconstructing the still image street scene, simulating crowd behavior, and rendering with consistent lighting, visibility, occlusions, and shadows. We demonstrate results on a diverse range of street scenes including regular still images and panoramas.

System Description

MY ALT TEXT

Our system has three major components. In Stage 1, we reason about the scene by predicting its semantic segmentation labels, depth values, sun direction and intensity, as well as shadow regions. We additionally determine walking and driving regions for adding pedestrians and cars (red straight lines: lane detection; blue points: origin and destination points). In Stage 2, we simulate the pedestrians in a 2D bird's eye view representation (BEV) of the scene, and simulate car movements with predicted lanes (four colors correspond to four predicted path, both in BEV and scene images). If there is a detected crosswalk, we also simulate the traffic behavior by controlling a traffic light. In Stage 3, we render the scene with the estimated lighting, shadows, and occlusions. The whole pipeline is automated.

Another Carousel

Poster

Video Presentation

div

BibTeX


        @inproceedings{shan2023asv,
          title={Animating Street View},
          author={Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz},
          booktitle={Proceedings of ACM SIGGRAPG Asia 2023},
          year={2023},
          doi = {https://doi.org/10.1145/3610548.3618230},
        }