High-quality streamable free-viewpoint video

High-quality streamable free-viewpoint video
Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, Steve Sullivan.
ACM Trans. Graphics (SIGGRAPH), 34(4), 2015.
Multimodal reconstruction of tracked textured meshes.
Abstract: We present the first end-to-end solution to create high-quality free-viewpoint video encoded as a compact data stream. Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.
Hindsights: Letting mesh connectivity change temporally allows capture of more general scenes, but makes it more difficult to create transitions between different motions; we address this problem in the subsequent paper Motion graphs for unstructured textured meshes.