Matt's Homepage

Stavros decided to make a little e-ink display device that showed his outlook calendar and could sit next to his main monitor. He seemed to have a decent, basic understanding of programming, but had some clever ways of getting around things he didn’t know – namely – using CoPilot and sample code to hack together what he needed. I think it’s a great read to show how you can work through problems in a very pragmatic way – without re-inventing the wheel.

In the end, he struggled through finding a good quality e-ink display, an SDK that let him display on it consistently (running into many bad SDK’s and ones that left lots of artifacts), getting calandar graphics on the device, and 3d printing the case it was mounted in.

Most interesting to me was that instead of trying to interface with his calendar app and go through the difficult work of re-creating a properly formatted/sized and good looking calendar graphics – he came up with a much more simple and easy method. He admits he wasn’t very good at C++ programming and had some false starts trying to find a software package that let him render consistently to the display. There were many that didn’t work right, left lots of lines on the screen, etc.

He then took his C++ compiler and a block of framebuffer rendering sample code. Then, with the help of CoPilot, he stumbled through a method that simply displaying the calendar in a web browser, copy the screen, download the image file over HTTP, and copy the bytes directly onto the framebuffer.

He set up a sever-side script to generate the image along with a hash of the image so the device knew when an actual update happened to the image since he didn’t want the e-ink display constantly flashing if it didn’t have a real update for the display.

A clever bit of hackery – and demonstrates how simply things can be made if you are creative.

Article:

• https://www.stavros.io/posts/making-the-timeframe/

Neural Radiance Fields (NeRF) produce some pretty beautiful renderings. A little like photogrammetry, it utilizes objects placed in a multi-dimensional volume (as captured from multiple viewpoints) and then when you want to render it from a particular angle, shoots rays into the scene based on camera location and queries the volume in order to get a screen coordinate pixel color at that location.

It does suffer from some shortcomings – such as largely only working well on static scenes, has trouble when there is missing or occluded portions, and most notably renders objects that lack fine details or produces blobby geometry common to volumetric rendering techniques.

But it doesn’t stop people from trying. Zip-NeRF is an example where these Google scientists demonstrate how ideas from rendering and signal processing yield better error rates and trains dramatically faster than previous techniques.

It’s always interesting to see what new things people are trying out these days.

Say hello to Gaussian splatting. It allows high quality, realtime rendering of scenes captured from multiple photos or videos.

Gaussian Splatting is a rendering technique that can produce extremely high quality rendering at very high frame rates. It uses a novel new technique who’s closest cousin is probably photogrammetry. Photogrammetry has been around for awhile (taking many 2D pictures of an object from many different directions and then re-building a 3D object). 3D Gaussian Splatting takes this much further.

Gaussian Splatting starts with lots of pictures like photogrammetry, but it then converts the data into a point cloud. The points become gaussians with are then used by the rendering routine.

1. Take a collection of photographs or extremely high quality renderings from a number of different camera positions all around the environment. The individual points from each of the photos becomes gaussians in 3D space.
2. The gaussians are not correct for rendering, so you must run a training pass over them much like a 1 layer neural net – but with special properties like densification and pruning.
3. From your camera position, projecting the gaussian points back into the 2D plane based on camera
4. Sort by depth
5. Iterate over each gaussian for a given pixel and sum the contribution.
6. This trained set can then be rendered from any angle.

Update 11/2023: There’s also a way of handling animated objects via 4D Gaussian splatting.

Articles:

• https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
• https://huggingface.co/papers/2308.04079