Come to the Table

This series is the story of a research project that kept refusing to end. It runs from a kitchen table in Auckland in 2019 to a demo booth in Sydney four years later, and along the way it picks up depth cameras, AR headsets, a shape memory alloy actuator that presses your nose, and a community of people who kept pulling the work forward. This first post is about where it began: SIGGRAPH Asia 2019 in Brisbane, and a project called Come to the Table! Haere mai ki te tēpu!

Mairi's question

The project came from Mairi Gunn, an artist and researcher at the Elam School of Fine Arts in Auckland. Her question was disarmingly simple: New Zealand was colonised in a way that left relations between Māori and the descendants of settlers in a difficult state, and most encounters between strangers from different cultures happen at a distance: in the street, guarded, brief. What would it take to imagine a healthy partnership instead? Her answer was furniture. A kitchen or dining table is where the real conversations in a home happen. So: invite people to the table.

My job, alongside Huidong Bai at the Auckland Bioengineering Institute, was to make the table work, technically. If the person across the table from you is a recording, they should still feel present: three-dimensional, life-sized, sitting in your actual space. That means volumetric capture, and in 2019 that meant building most of the pipeline ourselves.

Version one: a sensor totem on the table

The capture rig for the first version was a cluster of depth sensors arranged around a central 3D 360 camera, planted in the middle of a real table like a strange centrepiece. Before the conference, Māori participants sat around that table in New Zealand and simply had a conversation about culture and about being at the table of power, while the rig captured a full 3D recording of everyone present, along with spatial audio.

Playing it back was the part that still gets me. Point clouds are messy, noisy, honest things. You can see the seams and the sparkle of dropped depth samples, and yet when a recorded person turns and laughs across the table, none of that matters. Your brain files them under someone is here with me, not I am watching a video.

Taking it to Brisbane

At SIGGRAPH Asia 2019 the experience had three parts. Attendees sat at a physical table wearing an AR headset and found the recorded guests seated around it, mid-conversation. Then one of the virtual guests would turn to the visitor and ask what they thought. And here was the bit we were most proud of: the visitor's response was captured live, in 3D, by the same style of rig sitting on the conference table. At a second table nearby, other attendees in VR headsets could walk around those recorded responses and watch the conversation grow.

Plenty of projects had shown pre-recorded volumetric video by 2019. As far as we knew, this was one of the first to close the loop: capture the audience's reply and fold it back into the work, so the conversation kept accumulating strangers.

Not everything cooperated. Live 3D capture on a demo floor is a lesson in humility: lighting changes, Wi-Fi congestion, and the occasional attendee who leans directly into the sensor cluster. But people sat down guarded and stood up talking to us about their own families. That reaction is what convinced us the table deserved more iterations.

The paper is a short read: Come to the Table! Haere mai ki te tēpu! (SIGGRAPH Asia 2019 XR). And the underlying capture system I was building at the time is described in more detail in this post.

The next iteration swapped the crowd of guests for a single, quietly powerful encounter: one visitor, one volumetric Māori woman, one table. That became First Contact – Take 2, the next post in this series.