Unified Gaussian Primitives for Scene Representation and Rendering

Abstract

Searching for a unified scene representation remains a research challenge in computer graphics. Traditional mesh-based representations are unsuitable for dense, fuzzy elements and introduce additional complexity for filtering and differentiable rendering. Conversely, voxel-based representations struggle to model hard surfaces and high-frequency details. We propose a general-purpose rendering primitive based on 3D Gaussian distributions for unified scene representation, featuring versatile appearance ranging from glossy surfaces to fuzzy elements, as well as physically based scattering to enable accurate global illumination. We formulate the rendering theory for the primitive based on non-exponential transport and derive efficient rendering operations to be compatible with Monte Carlo path tracing. The new representation can be converted from different sources, including meshes and 3D Gaussian splatting, and further refined via transmittance optimization thanks to its differentiability. We demonstrate the versatility of our representation in various rendering applications such as global illumination and appearance editing, while naturally supporting arbitrary lighting conditions. With suitable simplification, we further adapt our method to radiance field reconstruction and rendering. We conduct comprehensive comparisons of our representation with existing scene representations, highlighting its efficiency in capturing details and representing aggregate elements.