{"id":9627,"date":"2023-10-19T14:02:04","date_gmt":"2023-10-19T21:02:04","guid":{"rendered":"https:\/\/mattfife.com\/?p=9627"},"modified":"2024-01-13T13:03:06","modified_gmt":"2024-01-13T20:03:06","slug":"gaussian-splatting-graphics-pipeline","status":"publish","type":"post","link":"https:\/\/mattfife.com\/?p=9627","title":{"rendered":"Gaussian Splatting graphics pipeline"},"content":{"rendered":"\n<p>Say hello to <a href=\"https:\/\/huggingface.co\/papers\/2308.04079\" data-type=\"link\" data-id=\"https:\/\/huggingface.co\/papers\/2308.04079\">Gaussian splatting<\/a>. It allows high quality, realtime rendering of scenes captured from multiple photos or videos. <\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/repo-sam.inria.fr\/fungraph\/3d-gaussian-splatting\/content\/videos\/bicycle.mp4\"><\/video><\/figure>\n\n\n\n<p>Gaussian Splatting is a rendering technique that can produce extremely high quality rendering at very high frame rates. It uses a novel new technique who&#8217;s closest cousin is probably photogrammetry. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Photogrammetry#:~:text=June%202019),radiant%20imagery%20and%20other%20phenomena.\" data-type=\"link\" data-id=\"https:\/\/en.wikipedia.org\/wiki\/Photogrammetry#:~:text=June%202019),radiant%20imagery%20and%20other%20phenomena.\">Photogrammetry <\/a>has been around for awhile (taking many 2D pictures of an object from many different directions and then re-building a 3D object). 3D Gaussian Splatting takes this much further.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<span class=\"embed-youtube\" style=\"text-align:center; display: block;\"><iframe loading=\"lazy\" class=\"youtube-player\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/HVv_IQKlafQ?version=3&#038;rel=1&#038;showsearch=0&#038;showinfo=1&#038;iv_load_policy=1&#038;fs=1&#038;hl=en-US&#038;autohide=2&#038;wmode=transparent\" allowfullscreen=\"true\" style=\"border:0;\" sandbox=\"allow-scripts allow-same-origin allow-popups allow-presentation allow-popups-to-escape-sandbox\"><\/iframe><\/span>\n<\/div><\/figure>\n\n\n\n<p>Gaussian Splatting starts with lots of pictures like photogrammetry, but it then converts the data into a point cloud. The points become gaussians with are then used by the rendering routine.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Take a collection of photographs or extremely high quality renderings from a number of different camera positions all around the environment. The individual points from each of the photos becomes gaussians in 3D space.<\/li>\n\n\n\n<li>The gaussians are not correct for rendering, so you must run a training pass over them much like a 1 layer neural net &#8211; but with special properties like densification and pruning.<\/li>\n\n\n\n<li>From your camera position, projecting the gaussian points back into the 2D plane based on camera<\/li>\n\n\n\n<li>Sort by depth<\/li>\n\n\n\n<li>Iterate over each gaussian for a given pixel and sum the contribution.<\/li>\n\n\n\n<li>This trained set can then be rendered from any angle.<\/li>\n<\/ol>\n\n\n\n<p><strong>Update 11\/2023<\/strong>: There&#8217;s also a way of handling animated objects via 4D Gaussian splatting.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<span class=\"embed-youtube\" style=\"text-align:center; display: block;\"><iframe loading=\"lazy\" class=\"youtube-player\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/BHe-BYXzoM8?version=3&#038;rel=1&#038;showsearch=0&#038;showinfo=1&#038;iv_load_policy=1&#038;fs=1&#038;hl=en-US&#038;autohide=2&#038;wmode=transparent\" allowfullscreen=\"true\" style=\"border:0;\" sandbox=\"allow-scripts allow-same-origin allow-popups allow-presentation allow-popups-to-escape-sandbox\"><\/iframe><\/span>\n<\/div><\/figure>\n\n\n\n<p>Articles:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/repo-sam.inria.fr\/fungraph\/3d-gaussian-splatting\/\">https:\/\/repo-sam.inria.fr\/fungraph\/3d-gaussian-splatting\/<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/huggingface.co\/papers\/2308.04079\">https:\/\/huggingface.co\/papers\/2308.04079<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Say hello to Gaussian splatting. It allows high quality, realtime rendering of scenes captured from multiple photos or videos. Gaussian Splatting is a rendering technique that can produce extremely high quality rendering at very high frame rates. It uses a novel new technique who&#8217;s closest cousin is probably photogrammetry. Photogrammetry has been around for awhile (taking many 2D pictures of an object from many different directions and then re-building a 3D object). 3D Gaussian Splatting takes this much further. Gaussian&#8230;<\/p>\n<p class=\"read-more\"><a class=\"btn btn-default\" href=\"https:\/\/mattfife.com\/?p=9627\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[9,5],"tags":[],"class_list":["post-9627","post","type-post","status-publish","format-standard","hentry","category-cool","category-technical"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p4WECr-2vh","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/9627","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9627"}],"version-history":[{"count":7,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/9627\/revisions"}],"predecessor-version":[{"id":10393,"href":"https:\/\/mattfife.com\/index.php?rest_route=\/wp\/v2\/posts\/9627\/revisions\/10393"}],"wp:attachment":[{"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9627"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9627"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mattfife.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9627"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}