The Sensor Was Already in the Headset

Every study in this series so far involved putting something on somebody. A Shimmer3 unit with electrodes on two fingers. An EEG attachment clipped to the front of a headset. A chest strap. Each one is a thing that needs charging, a thing that slides out of position, a thing that produces artifacts when the wearer scratches their nose, and a thing that has to be explained and consented to before a study can start.

That is me in the lab, wearing a g.tec fNIRS cap under a Magic Leap 2, in front of a mirror display that was showing my own signals back at me. It worked, and it is also completely unshippable. Nobody is doing that in their living room.

There is a much older literature that avoids all of this. Researchers have been inferring stress and cognitive load from keystroke timing and mouse movement for years, on the theory that affect and cognitive load change how you control your muscles, so they leak into whatever you are already doing with your hands. No extra hardware. Less identifying than a face or a voice, which matters if you ever want to ship it.

VR headsets now track hands at high fidelity as a standard input method. So the question is obvious in hindsight: is the input itself the signal?

This study was led by Phoebe Chua, with Yadeesha Weerasinghe and Suranga Nanayakkara, and I worked on it as second author.

The study

Twenty-two participants, ranging from people who had never worn a headset to people using VR several times a week, doing four free-hand tasks at two difficulty levels each, plus a calming baseline environment. Card sequences, button sequences, UI navigation, and a slingshot. Around two and a half hours of hand and head tracking data in total.

From the raw tracking we pulled features describing how a gesture is formed rather than which gesture it is. Speed. Distance travelled. Hand tension, meaning how tightly the fingers are held during the pinch. Head motion.

What the hands give away

The associations came out clearly. Reported valence, arousal and cognitive load were all linked to significant differences in gesture speed and distance. On harder tasks people held their hands more tensely and moved their heads less.

The head result is the one I did not expect and like the most. Under load, people go still. It reads as concentration narrowing the body down, and it is visible in a signal every headset already logs continuously for the purpose of drawing the correct frame.

For prediction we used support vector classifiers, deliberately simple, as a baseline rather than a bid for state of the art. Within a single user, separating the easy from the challenging version of a task ran at 85 to 91% accuracy depending on the task, with F1 scores above 0.80. Recognising affect and cognitive load directly was much more variable, from 87% at best down to 50%, which is chance.

The catch

Then you switch to leave-one-user-out validation, where the model is trained on everyone else and tested on a person it has never seen, and task condition accuracy falls to roughly 0.43 to 0.45.

That is the whole story of the paper in one number. Within a person, hand motion is a reliable tell. Across people, a simple model of it mostly collapses. It learns your hands, not hands.

There is one interesting exception. On the slingshot task, arousal and valence held up across users at around 0.86 to 0.88. My reading is that the slingshot constrains the movement more than the other tasks do, so everyone performs roughly the same gesture and individual style has less room to drown out the signal. If that is right, then generalisation is partly a design problem rather than purely a modelling one, and you get it by choosing interactions with tighter motor envelopes rather than by collecting more data.

Where this leaves the series

Three years, roughly, from arguing that headsets should read the body, to building a full physiological rig and watching people ignore it, to finding usable signal in a data stream the device already produces for free.

The sensing question has a reasonable answer now. Yes, you can tell. Not perfectly, not yet across strangers, but the direction is clear and it needs less hardware than I assumed in 2022.

The question I cannot answer with any of these papers is what a system should do once it knows. The assembly study showed that displaying the state to a human is close to useless on its own. That leaves acting on it automatically, which turns out to have its own cost: in a later study on AR-assisted construction, taking control away from trained carpenters reduced their mental workload and reduced their sense of autonomy at the same time. Comfort and agency, traded against each other, with the system holding the dial.

That is the problem I am still working on, and it is a design question at least as much as a sensing one. The next post is the year I spent attacking it from three directions at once, and Zenflow is the clearest answer I have managed so far. It works precisely because it never tells you anything. It just changes the room around you, slowly, while you breathe.

The paper is Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR, by Phoebe Chua, Prasanth Sasikumar, Yadeesha Weerasinghe and Suranga Nanayakkara.