High-quality streamable free-viewpoint video
ACM Trans. Graphics (SIGGRAPH), 34(4), 2015.
Multimodal reconstruction of tracked textured meshes.
We present the first end-to-end solution to create high-quality free-viewpoint video encoded as a compact
data stream. Our system records performances using a dense set of RGB and IR video cameras, generates
dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances
contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette
information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to
create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video
stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding
efficiency. We release several datasets with calibrated inputs and processed results to foster future
Letting mesh connectivity change temporally allows capture of more general scenes,
but makes it more difficult to create transitions between different motions;
we address this problem in the subsequent paper
Motion graphs for unstructured textured meshes