Nobody Looked at the Heart Rate

Two years earlier I had written a paper arguing that collaborative VR should share physiological signals. Heart rate, cognitive load, attention. The pitch was that remote collaborators lose the small involuntary cues you get for free when you sit next to someone, and a headset already has enough sensors to put some of them back.

It was a position paper. No system, no study, just an argument. This is the paper where we finally had to find out whether the argument was right.

Building the thing I had argued for

The system runs in Unity across two HTC Vive Pro Eye headsets on a local network, using Ubiq for the networked social VR layer. Each person wears a Shimmer3 sensor for galvanic skin response and heart rate, with a Looxid EEG attachment on the headset providing an attention value. Everything streams into Unity over Lab Streaming Layer.

Three cues, three visual treatments. Heart rate is an animated heart that beats with your pulse and shows the number. Cognitive load is a brain icon on a low, medium and high traffic-light scale, derived from GSR. Attention is a moving circle showing a percentage, from the EEG.

The bit I still like is the calibration. Cognitive load means nothing as an absolute GSR number, so before each session participants ran one-back and two-back tests, which are standard working memory tasks with a known difficulty gap. That gives you a personal low, medium and high to scale the traffic light against, rather than a threshold borrowed from someone else's body.

The task was engine assembly. A virtual engine with more than 50 separable parts on a stand in a replica workshop, with the right tools laid out around it, and twelve tasks graded easy, medium and hard, from removing flywheel bolts up to the crankshaft bearing. One person holds the instructions, the other does the work, and they swap halfway with the difficulty balanced so nobody gets the easy half twice.

Here is the whole thing running, with the cues live:

Twenty-eight participants in fourteen pairs, four conditions, within subjects. Heart rate alone as the base, then heart rate plus cognitive load, heart rate plus attention, and everything at once.

What happened

Task completion time: no significant difference across the four conditions. Workload on the NASA-TLX: no significant difference. The hypothesis I had written in 2022, that seeing these cues would improve performance, simply did not hold.

Two things did move. Co-presence differed significantly between conditions, and attention allocation differed strongly. When asked to rank the four conditions, 15 of 28 people put the attention cue first, and more than half preferred it overall.

Then we looked at the eye tracking, which is the number that has stayed with me.

Participants looked at their partner's physiological cues for less than 10% of the session. For the largest group of them it was between 1% and 5%. Around 70% of the looking that did happen occurred in the first minute.

That is a habituation curve. The cues were novel, people checked them, and then they got on with the engine.

Why I think it failed

The paper offers several explanations and I believe all of them, but they are not equally interesting.

• The task was wrong, and that is on me. Engine assembly is procedural. Your partner's stress level does not change which bolt comes off next. We chose a physical task because it is realistic for training, but knowing someone is overloaded only matters if you can act on it. A negotiation or a design decision would have given the cues something to do.
• Heart rate was in every condition. That was a design error. Participants could read stress off the heartbeat alone, so the conditions were never as separated as the study design implied.
• The cues disagreed about time. Heart rate updated in real time. The Looxid attachment averages cognitive load over a 10-second epoch. So two numbers sat side by side in the same visual language, one describing now and one describing ten seconds ago, with nothing telling the user which was which. If you are going to fuse signals, they need a shared clock, and ours did not have one.
• People did not know what to do with the information. This is the finding that mattered most. Participants said outright that they struggled to interpret their partner's signals, and several said they would rather the system made the judgment for them. One wanted to see everything and choose what was relevant per task. Another said processing it all was simply too much.

The thing worth taking away

I had spent two years assuming the hard part was capture. Get the sensors on, get the signals clean, get them into the shared scene at low latency. We did all of that and it worked, and it changed nothing about how well two people assembled an engine.

The hard part is interpretation. A number on a wall is not a cue, it is homework. The participants who asked for the system to do the judging were describing the actual research problem, and every system I have worked on since has been some version of trying to answer them.

The one genuinely encouraging result is the attention cue. It was preferred, it moved co-presence, and I think that is because it is the only one of the three that answers a question people are already asking during collaboration, namely whether the other person is still with you. Heart rate and cognitive load answer questions nobody was asking yet.

Worth stating plainly: this study did not show that sharing physiological cues improves collaboration. It showed that we do not yet know how to present them so that anyone would use them.

The paper is A User Study on Sharing Physiological Cues in VR Assembly Tasks, IEEE VR 2024, with Ryo Hajika, Kunal Gupta, Tamil Selvan Gunasekaran, Yun Suen Pai, Huidong Bai, Suranga Nanayakkara and Mark Billinghurst.