Four years after the first table in Brisbane, the project came back to SIGGRAPH Asia, this time in Sydney, on a Magic Leap 2, with a capture pipeline light enough to fit in a suitcase. The final post in this series, with a look back at how far volumetric capture came between 2019 and 2023.
This project started at SIGGRAPH Asia 2019 in Brisbane with a table full of depth sensors. Four years later it returned to SIGGRAPH Asia, now in Sydney in December 2023, as Haptic Hongi – Reiterated. Same kaupapa, same table, same Tania welcoming strangers. Almost everything underneath had changed.
What "reiterated" meant in practice
Two big upgrades defined this version.
The capture pipeline got radically simpler. In 2019 we were wrangling a cluster of depth sensors and a home-grown pipeline. This time, Tania was recorded with a single Azure Kinect and encoded through DepthKit. We had experimented with the Intel RealSense and Azure Kinect SDKs for compression and playback, and DepthKit hit the best balance between file size and smooth playback: a steady 30 frames per second from hardware that fits in a suitcase. A single-camera rig would normally be a limitation, but the table-top format quietly absorbs it: the visitor sits directly opposite Tania, and she is always gazing straight at them, so the one angle you have is the one angle that matters.
The headset changed. The HoloLens 2's 43-degree diagonal field of view had always cropped Tania when she leaned in close, which is a bad moment to lose her. The Magic Leap 2 brought a taller field of view, a slighter profile, and a party trick called segmented dimming that darkens the world exactly where the virtual content sits, making Tania appear genuinely solid at the table instead of ghost-like.
The haptic unit, the sponge-tipped shape memory alloy actuator from the previous post, carried over, remounted for the new headset, still pressing your nose and forehead at the exact moment Tania's hongi lands.
The booth
A SIGGRAPH XR demo is a strange kind of theatre: you build a domestic dining room inside a black-curtained expo hall, and then you defend it against convention-centre Wi-Fi for four days.
The queue conversations were the best part. You could watch people arrive curious about the headset and leave talking about first encounters, colonisation, and who gets a seat at whose table. Visitors from migrant backgrounds often seemed relieved at how warmly Tania greeted them. Māori visitors told us, more than once and to our surprise, that they could feel the wairua, her spirit, across time and space, and that the actuator's gentle pressure accentuated it. For a system whose entire purpose is presence, there is no better review.
Four years in ninety seconds
Standing at that booth, the whole arc of this series kept compressing itself into the minute-and-a-half we had to explain the project to each visitor:
- 2019, Brisbane: a cluster of depth sensors on a table, recording conversations between Māori participants and capturing visitors' replies live. Presence as a group phenomenon.
- 2020: First Contact – Take 2. One visitor, one volumetric woman, one table. Presence as an encounter.
- 2022: Haptic Hongi. A shape memory alloy actuator gives the encounter touch. Presence you can feel on your skin.
- 2023, Sydney: the same encounter, sharper and more solid, on hardware light enough to take anywhere.
The pattern across those four years is the one I find most encouraging about this field: every iteration, the technology got smaller and the experience got bigger. We went from eight sensors to one, from a custom pipeline to an off-the-shelf tool, from a headset that cropped the moment to one that dims the world around it, and none of the effort saved went to waste, because all of it moved into the encounter itself.
I put together a video tracing this whole evolution, from the rigs to the playback to the hongi:
The Sydney iteration is written up in Haptic Hongi – Reiterated (SIGGRAPH Asia 2023 XR). There is more to do; eye-tracking so Tania can make true eye contact is high on the list. If the last four years are any guide, the table will find its way to another room, in another city, and someone new will sit down at it.
Ngā mihi nui to Mairi Gunn, who has driven this waka from the start, to Tania Remana, whose performance is the heart of every version, and to Sachith Muthukumarana, Mark Billinghurst, and Huidong Bai.