"Light" plays a crucial role in film and television works, games and virtual environment. It is an important modeling method, and plays a role in conveying information, expressing emotions, setting off the atmosphere, portraying characters' personalities and psychological changes.
It influences the formation of the tone of the film and the presentation of the film style, forms a unity of opposites with the tone of the film, and combines with other modeling means to show the rhythm and melody of the film.
Many special effects scenes in film and television works need to be completed with the help of green screen, when the technology is not too good, often occur "50 cents special effects" of the massacre.
Copying the perfect light effect is even more difficult.With the evolution of computer vision technology, computers have been able to "naturally" restore face shapes and skin patterns, but there is still a lack of realism in simulating lighting conditions.And Google recently announced a new technology based on lighting, Relightables.It's a perfect solution to that.The Relightables system workflow can be divided into three parts: capture, rebuild, and render.The core of the system relies on a spherical lighting device containing a multi-angle (active) stereo depth sensor, which is fitted with 331 programmable lights and 90 high-resolution 12.4MP reconstruction cameras.
The cameras used to capture the human body include 32 infrared (IR) cameras and 58 RGB cameras. Infrared sensors provide accurate and reliable 3D data, and RGB cameras capture high-quality geometric normal maps and textures. These cameras record raw video at 60Hz, and researchers alternate between two different lighting conditions based on spherical gradient illumination.
Next, the researchers upload the data to a public repository. The first stage is to generate a depth map, segmentation map, and 3D mesh for each "camera" [Kazhdan and Hoppe 2013].
They use an alignment algorithm to process the sequence of the reconstructed mesh. In this way, long subsequences can share common triangulation. Researchers have proposed a new method to solve the key frame selection problem and turn it into an MRF inference problem to solve. Each unique triangulation is parameterized into a normal 2D texture space, which can be shared with all frames that share the triangulation.
Finally, there are two gradient spherical illumination images available for each grid, from which albedo, normal, gloss, and ambient light occlusion maps can be generated. These maps are compatible with standard rendering engines and can be used to regenerate rendered images under any set lighting conditions.
Google's new system can perfectly restore the light and shadow effects around people, making the composite image look more realistic. The captured human body can be seamlessly fused into digital scenes in the real world or in movies, games, etc. It could revolutionize the field of 3D capture technology.