How the Gaussian splatting viewer works

A 3D Gaussian splatting scene is not a mesh. It is a few hundred thousand translucent ellipsoids floating in space, each with a position, a covariance matrix describing its shape and orientation, a colour and an opacity. Render them in the right order and you get a photograph you can walk around inside. Render them in the wrong order and you get fog.

That last sentence is the whole engineering problem. Everything else — reading the file, packing the data, projecting the ellipsoids — is bookkeeping. The ordering is what makes it hard, and it is what the viewer spends most of its budget on.

Why order matters so much

Splats are blended, not depth-tested. A solid triangle can be drawn in any order because the depth buffer discards whatever is behind it. A translucent splat has to be composited over whatever is behind it, and alpha compositing is not commutative: red over blue is a different pixel than blue over red. So every splat has to be drawn strictly back-to-front from the camera's point of view.

Which means: every time the camera moves, the entire scene has to be re-sorted. Not culled, not partially updated — re-sorted, hundreds of thousands of elements, every frame you turn the mouse. Do that on the main thread and the page stops responding to input, which is precisely when the user is dragging.

Sorting off the main thread

The viewer pushes the sort into a Web Worker. The main thread owns the camera and the draw call; the worker owns the positions and the ordering. Each frame the main thread hands the worker the current view matrix; the worker computes a depth per splat, sorts the indices, and posts back the ordered index buffer as a transferable — no copy, just a pointer handoff.

The sort is a counting sort, not a comparison sort. Depth is a single dot product against the camera's forward axis — the third row of the view matrix — and those depths get quantised into 65,536 buckets between the frame's min and max. Count the occupants of each bucket, prefix-sum the counts into start offsets, then scatter each index into its slot:

var b = ((dep[i] - minD) * range) | 0;   // quantise
counts[b]++;                             // count
starts[i] = starts[i-1] + counts[i-1];   // prefix sum
order[starts[bucket[i]]++] = i;          // scatter

That is O(n) with three linear passes instead of O(n log n) with a comparator. For this workload — many elements, no need for stability, and a depth precision far finer than the eye can resolve — the quantisation is invisible and the constant factor is what actually matters. Ascending depth order means farthest-first, which is exactly the back-to-front order over-compositing requires.

Because the worker runs behind the render loop rather than inside it, the display can be one sort behind the camera during a fast drag. You do not notice. A frame ordered for where the camera was 16ms ago is far less objectionable than a frame that arrives 80ms late.

Packing covariance into a texture

Each splat needs its 3D covariance available in the vertex shader. Passing that as vertex attributes would mean a large interleaved buffer and a lot of per-instance bandwidth. Instead the worker packs everything — position, colour, and the covariance terms — into a single usampler2D, an unsigned integer data texture, and the shader fetches what it needs by index:

uvec4 cen = texelFetch(u_texture, ivec2((uint(index) & 0x3ffu) << 1, uint(index) >> 10), 0);
uvec4 cov = texelFetch(u_texture, ivec2(((uint(index) & 0x3ffu) << 1) | 1u, uint(index) >> 10), 0);

The bit-twiddling is a 1024-wide texture addressing scheme: the low 10 bits of the index are the column, the rest are the row, and each splat occupies two adjacent texels — one for centre and colour, one for covariance. Integer textures matter here because the covariance terms are packed as half-floats two-per-32-bit-word; sampling them as floats would let the GPU interpolate values that must not be interpolated.

Drawing: four vertices, many instances

The geometry is a single quad — four vertices, one triangle strip — drawn once per splat with instancing. The vertex shader reads the instance's covariance, projects it into 2D (a 3D Gaussian seen through a pinhole camera is a 2D Gaussian, which is the property the whole technique rests on), and stretches the quad to cover the resulting ellipse. The fragment shader evaluates the Gaussian falloff and writes colour × opacity.

The projection and covariance maths follow the MIT-licensed antimatter15/splat viewer, which is the clearest reference implementation of this step I know of.

What it doesn't do

There is no level-of-detail and no frustum culling either — every splat in the file is sorted and drawn every frame, so a very large scene brings your GPU down rather than degrading gracefully.

The .ply path also assumes the Inria training layout specifically. It requires x, y, z, scale_0..2, rot_0..3, opacity, f_dc_0..2 by name, un-logs the scales, pushes opacity through a sigmoid, and reorders the quaternion — Inria writes w, x, y, z where the packing wants x, y, z, w. Get that reorder wrong and every ellipsoid is rotated into a different wrong orientation, which looks like static rather than like a bug.

A small demo scene is embedded in the page as base64 so the viewer is never empty. It doubles as the fixture the renderer is checked against — if a change breaks the projection maths, the demo scene visibly deforms, which is a cheaper regression test than it sounds.