Which Cue Is Actually Doing the Work?

The previous post ended with an unanswered question. We had compared two bundles of communication cues, natural against device-centric, and learned that people's preferences split by role. What we could not say was which individual cue mattered. A bundle winning tells you nothing about its ingredients.

This is the study that took that apart properly. It was led by Huidong Bai, with Jing Yang and Mark Billinghurst, and I worked on it as second author. It ended up at CHI 2020, and it is the paper the rest of this series keeps leaning on.

A bigger, better rig

The system is a scaled-up relative of RemoteFusion. Where that used four RealSense D435s, this uses eight D415 cameras arranged in a sensor cluster, positioned by a field-of-view calculation so their frustums merge into a semi-sphere with no gaps.

The cluster, and the calibration solution showing all eight frustums merged.

Eight cameras into one machine is not a trivial ask. They ran over a PCIe USB 3.0 expansion board with extension cables, at 848×480 depth resolution and 30 FPS, deliberately below the sensors' maximum to keep the compute and network load survivable. After stitching, the effective rate settled at 20 FPS. Calibration was a one-time job using a MATLAB multi-camera toolbox.

The local room is reconstructed and streamed; gaze and gesture come back the other way.

The local worker wore a Magic Leap One. The remote expert wore an HTC Vive Pro Eye, which has an eye tracker built in, with a Leap Motion mounted on the front plate for hand tracking. Point clouds were compressed with Google's Draco library and pushed over a local network, landing under 300 ms end to end.

The alignment trick is worth noting because it is the same problem as the phantom overlay: an image marker sits in the centre of the workspace and acts as the shared origin. The Magic Leap finds it by image tracking, the Vive finds it via Lighthouse, and both systems agree on where zero is to within a centimetre or two.

Four conditions, not two

The experimental design is the entire point. Rather than two bundles, four conditions:

1. Verbal only (control)
2. Eye gaze alone, drawn as a purple raycast line from the expert's eyes
3. Hand gesture alone, drawn as a grey 3D hand mesh
4. Both combined

The same two cues as the expert sees them in VR, and as the worker sees them in AR.

There is a real conceptual distinction buried in those conditions. Gaze is implicit: it streams continuously and costs the expert nothing, because they are going to look at things anyway. Gesture is explicit: it requires the expert to decide to reach out and point. One is a side effect, the other is an act. That distinction turns out to matter.

The task was Lego bricks again, but harder. Twenty bricks in five colour groups of four, all same-coloured bricks grouped together with the same starting orientation so you could not just say "the red one." Twenty decoy white tags scattered among the real targets. Four L-shaped tables in a 7×5 m cubicle, with the camera cluster on the ceiling at 2.4 m. Each trial: find four bricks, carry them across, align each to a numbered tag's orientation.

The task as the worker saw it, and as the expert saw it in the reconstruction.

Twenty-four participants in twelve pairs, within-subject, counterbalanced, about an hour per pair.

What came out

Speed. Only the combined condition beat verbal-only by a statistically significant margin: 139 seconds against 228. Gaze alone (179 s) and gesture alone (173 s) landed in between and were not significantly different from anything. So the cues help, but individually neither is decisively better than the other.

Preference. Combined was the clear favourite, ranked first by 16 of 24 participants. Then gesture, then gaze, then control.

The interesting bit. Four local workers described the eye gaze as redundant or actively distracting, because the hand gestures already told them everything they needed. On system usability, gesture scored significantly higher than gaze on the local side.

That is not what the literature would have predicted. Gaze sharing had a lot of momentum as an idea, and it is seductive: free, continuous, no effort from the expert. But free and continuous also means unfiltered. A hand that points at something is a hand that decided to point at something, and that act of selection is doing a large part of the communicative work. A gaze ray shows you where the eyes went, including all the glances that meant nothing.

Six workers said the combined condition felt like being face to face with the expert, which is the sentence you actually want out of a study like this.

What I took from it

The finding I carried forward is that intentionality beats bandwidth. It would have been easy to conclude "more cues are better" and keep stacking channels. The honest reading is narrower: combining an implicit cue with an explicit one helps, gaze on its own is weak, and the reason is that explicit cues carry a decision along with the data.

It also set up the next question. Both cues here are visual, which means both compete for the same attention and both fail the moment the target is behind you. That is where spatial audio comes in, and it is the next post in this series.

The paper is A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing, CHI 2020, by Huidong Bai, Prasanth Sasikumar, Jing Yang and Mark Billinghurst. Figures above are from the paper.