Three Hypotheses and No Data

Every remote collaboration system I had built up to this point shared cues that people send on purpose. A reconstructed room. Gaze rays and hand meshes. A voice placed in 3D. Each one takes something you are already doing deliberately and moves it across the network.

Sitting next to somebody gives you a second channel that none of those cover. You can hear their breathing change. You notice when they go still. You catch the half-second pause before they answer, and you adjust without deciding to. None of it is chosen and all of it is information, and a video call throws essentially all of it away.

Headsets are unusual in that they already sit against the body. If you are willing to add a wrist or finger sensor, you can read heart rate, skin conductance and a rough attention signal in real time. So the argument of this paper was simple: put those in the shared space and collaborators will understand each other better.

What I built

PSCVR ran in Unity 2021 across two HTC Vive Pro Eye headsets on a local network. Ubiq handled the social VR layer, chosen over the alternatives because it was open source and readable rather than because it was the fastest. A Shimmer3 sensor gave galvanic skin response and heart rate. A Looxid attachment clipped to each headset supplied an attention value from EEG.

The task was a virtual engine with more than 50 separable parts, mounted on a stand in a replica workshop with the right tools laid out around it, built on the volumetric playback work from earlier in my PhD. Instructions appear, you find the tool, you take the part off.

Two details from the build that I still think were right.

The first is calibration. A GSR reading means nothing as an absolute number, so before each session participants would sit with a black screen for a minute and then run one-back and two-back tests. Those are standard working memory tasks with a known difficulty gap, which gives you a personal low, medium and high to scale against instead of a threshold borrowed from somebody else's body.

The second is that questionnaires were answered inside VR, between conditions, without removing the headset. Taking a headset off breaks EEG electrode contact and disturbs the GSR sensor, and then your next condition is measuring a slightly different person.

The part that was missing

There was no study.

The paper proposes four conditions: heart rate alone as the control, then heart rate plus cognitive load, heart rate plus attention, and all three together. It proposes a set of measures. It lists use cases in teaching, research tooling and counselling. And it states three hypotheses:

• Knowing your partner's physiological cues improves performance.
• Knowing your partner's physiological cues improves co-presence.
• Knowing your partner's physiological cues improves immersion.

That is where it ends. It went to the adjunct track at ISMAR 2022 as a system and a plan, which is a legitimate thing to publish, and I want to be straightforward about what it means: I had built the apparatus and written down what I expected, and I was arguing for a position I had not tested.

I believed all three hypotheses. They felt close to obvious. If you can see that your partner is overloaded, of course you will work better together.

What I would tell myself

It took until 2024 to run that study properly, and the answer to the first hypothesis was no. Not a small effect, not a nearly. No significant difference in task completion time, and no difference in workload either. The second hypothesis held up. The third never really got tested.

The gap between writing a hypothesis and being allowed to find out is much wider in research than people outside it assume, and two years is not unusual. What I would tell 2022 me is not that the idea was wrong, because the sensing worked fine. It is that I had spent all my design effort on capture and none on the question of what a person is supposed to do with a number that appears next to their colleague's head.

The paper is PSCVR: Physiological Sensing in Collaborative Virtual Reality, ISMAR Adjunct 2022, with Yun Suen Pai, Huidong Bai and Mark Billinghurst. The same year I also wrote a doctoral consortium paper at IEEE VR laying out the whole thesis, which is a similarly confident document written before most of the evidence existed.