Say hello to Gaussian splatting. It allows high quality, realtime rendering of scenes captured from multiple photos or videos.

Gaussian Splatting is a rendering technique that can produce extremely high quality rendering at very high frame rates. It uses a novel new technique who’s closest cousin is probably photogrammetry. Photogrammetry has been around for awhile (taking many 2D pictures of an object from many different directions and then re-building a 3D object). 3D Gaussian Splatting takes this much further.

Gaussian Splatting starts with lots of pictures like photogrammetry, but it then converts the data into a point cloud. The points become gaussians with are then used by the rendering routine.

1. Take a collection of photographs or extremely high quality renderings from a number of different camera positions all around the environment. The individual points from each of the photos becomes gaussians in 3D space.
2. The gaussians are not correct for rendering, so you must run a training pass over them much like a 1 layer neural net – but with special properties like densification and pruning.
3. From your camera position, projecting the gaussian points back into the 2D plane based on camera
4. Sort by depth
5. Iterate over each gaussian for a given pixel and sum the contribution.
6. This trained set can then be rendered from any angle.

Update 11/2023: There’s also a way of handling animated objects via 4D Gaussian splatting.

Articles:

• https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
• https://huggingface.co/papers/2308.04079