A short-throw projector and a depth camera turn a box of sand into a live topographic map. Built for Te Papa during the Mahuki accelerator, out of timber from my weekend job at PlaceMakers.
For about a year I worked at JIX, a small XR outfit whose tagline was "Joyful Immersive Experience" and whose actual pitch, printed on a card that sat on the desk behind me, was "sensory experiences for collection objects in GLAM institutions using mixed reality." GLAM is galleries, libraries, archives and museums. The short version: help museums let people touch things they are absolutely not allowed to touch.
That got us into Mahuki, the innovation accelerator run out of Te Papa, the national museum of New Zealand. Which meant a desk in the museum, access to their workshop, and a demo day at the end where we had to show something real to a room full of people who look at exhibits professionally.
I built a few things during that stretch. The one I still think about is the sandbox.
What it does
A wooden box holds a few centimetres of fine sand. Above it, on a post, sit two things: a short-throw projector pointing down, and a Kinect depth camera pointing at the same patch of sand.
The Kinect reads the height of the sand, continuously. That height map gets mapped through a colour ramp and a set of contour lines, and the projector paints the result back onto the sand it just measured. Dig a hole and it fills with blue. Pile sand up and the contour rings tighten around a peak and the colour shifts through green to brown. Drag a trench between two hills and a valley appears in the map before your hand has left the sand.
The whole thing is a closed loop that runs faster than you can move your hands, and that is the entire trick. There is no interface. Nobody needs instructions. You put your hands in sand, which every human already knows how to do, and the map keeps up.
The part I did not anticipate is how long people stay. Adults who came over to be polite were still there ten minutes later, building dams.
Building the thing
The software is the part people ask about. The part that actually took my time was the box.
I was working part time at PlaceMakers, which for anyone not in New Zealand is the big building supplies chain, the place tradies go for timber and fixings. The staff discount is why this project has a solid table under it instead of a repurposed desk. I bought the ply and the framing timber at a price I could justify on a startup salary, and carried it in.
Te Papa's workshop is a proper workshop: table saw, drill press, the whole thing. I had done enough carpentry to be useful there, and it turned out to be the most satisfying stretch of the whole accelerator. Marking out, cutting, checking square, doing it again because it was not square.
There were real constraints driving the dimensions. The projector is short-throw but not magic, so the height of the post above the sand is set by how much sand surface you need to cover. The Kinect has a minimum range, so it cannot sit too low either. The box has to be deep enough for people to actually shape sand and shallow enough that a child can reach the middle. And the whole assembly has to roll, because it is a museum and everything has to move.
The alignment between projector and depth camera is the fiddly bit, and it is not a thing you solve once. Both are looking at the same surface from slightly different positions, so you have to work out the mapping between what the camera measures and where the projector paints. Get it wrong by a couple of centimetres and the contours lag behind your hands in a way that feels subtly broken even to people who could not tell you what is wrong.
Why a museum wanted one
The sandbox on its own is a nice toy. What made it a pitch was pointing it at real terrain.
On demo day we had a board up showing what the same rig does with actual New Zealand geography loaded in: Wellington's port terrain against sea level, the Kaikōura coastal road that got destroyed and rebuilt after the 2016 earthquake, Mount Ruapehu with a simulated eruption. The sandbox is the interactive front end for a conversation about landscape, and in a country that is visibly shaped by earthquakes and volcanoes, that conversation is not abstract.
It also got written up, which was a first for me: seeing a thing I had built with a table saw show up in print, next to a clean little diagram labelling the short-throw projector and the 3D camera.
The other one
The sandbox was not the only build. The other piece I liked was a virtual haptics station: a Leap Motion sensor under a screen, letting you reach out and turn over a collection object with your bare hands.
The point of it was objects that were not in the building. Museums hold far more than they display, and plenty of what they hold is too fragile, too sacred, or simply somewhere else. Being able to pick one up and rotate it, with your own hands and no controller, is a different proposition from looking at a photo of it.
It was rougher than the sandbox. Leap Motion tracking of that era dropped your hands the moment they occluded each other, and the lack of any physical resistance meant people kept pushing their hands through the object. The sandbox worked better precisely because the physical thing was real, and the digital layer only had to describe it.
The rest of it
Accelerators are mostly the people. The JIX crew and the wider Mahuki cohort ended up being the reason that year was fun rather than just busy.
Looking back, the sandbox is the clearest example I have of something I keep relearning: the most convincing interactive work is often the work where the computer does the smallest possible amount. No headset, no controller, no calibration ritual for the visitor. A box of sand, and a projector that keeps up.