Animating Street View

In a groundbreaking study titled "Animating Street View," researchers Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, and Steve Seitz have developed a system that brings static street view images to life by populating them with naturally behaving, animated pedestrians and vehicles.

This innovative system operates by first removing existing people and vehicles from the input image. This is achieved through segmentation and inpainting using the Stable Diffusion tool. Once the scene is cleared, the system inserts moving objects with proper scale, angle, motion, and appearance. This is managed by reconstructing the still image street scene, simulating crowd behavior, and rendering with consistent lighting, visibility, occlusions, and shadows.

A significant aspect of this system is its capability to plan paths and traffic behavior. Using potential-field based gradient descent, the system enables real-time path planning and obstacle avoidance. Traffic behavior is simulated by controlling the state of each crosswalk, creating a traffic-light interaction model where pedestrians and cars wait in front of the crosswalk when the light is red.

Rendering the scene with consistent lighting, visibility, occlusions, and shadows is another challenge tackled by this system. By leveraging existing information from pre-trained models such as depth, segmentation, and shadow detection, the system can render consistent lighting and visibility. It also employs a 3D shadow occluder to cast shadows onto inserted objects, enhancing the realism of the scene.

The system was tested on a diverse range of street scenes, including regular still images and panoramas. The results were impressive, demonstrating the system's ability to produce arbitrarily long videos with moving people and cars, adding a dynamic dimension to static images. This research opens up new possibilities in the field of image animation and has potential applications in various domains, from entertainment and gaming to urban planning and traffic management.

Read the whole article here: http://arxiv.org/abs/2310.08534v1