A VR photo walk on a virtual island, where the sky tints red when you get stressed and the companion beside you changes its voice. Two demos, one headset with the sensors built in, and a shift from showing your state to another person to letting the world respond to it directly.
The previous system put your physiological state in front of another person and hoped they would make something of it. These two projects take the other road entirely. The system reads you, and instead of reporting on you, it changes the world you are standing in.
They were led by Kunal Gupta at the Empathic Computing Lab, and I worked on them as a co-author. I am writing them up together because they are two versions of the same idea a year apart, and the difference between them is the interesting part.
One headset instead of a pile of devices
PSCVR needed a Vive, a Shimmer unit strapped to a hand, and a Looxid attachment clipped onto the front of the headset. Three vendors, three SDKs, three things to fail.
Both of these run on the Galea, which is OpenBCI's sensor package built into a Varjo Aero. At the time it had the widest range of physiological sensors integrated into a single display: eight EEG electrodes across the top and back of the head over the visual cortex, two more in the faceplate, plus four EMG, two EDA, four EOG and a PPG sensor, with eye tracking at 200 Hz.
This matters more than it sounds. In the assembly work, half the fragility of a session was in the fact that the sensors were bolted on afterwards. When the sensing is part of the display, a study stops being a hardware wrangling exercise and becomes something you could imagine shipping.
VRdoGraphy, the first version
The demo we took to IEEE VR in early 2023 was a VR photography experience. EEG, EDA, EMG and HRV stream into a live emotion prediction model, and two things respond to its output: the environment and a companion.
The environment shifts colour. The companion, an abstract circle attached to your viewpoint, changes both its colour and the tone of its voice when it speaks to you.
That is the whole idea, and it already feels different from displaying a heart rate. Nobody has to read anything. If the estimate is right, the world simply feels like it is going along with you.
SensoryScape, and the thing that got added
The version we took to SIGGRAPH Asia in Sydney at the end of the year added context, and context turns out to be the upgrade that matters.
Knowing that someone is stressed is not very actionable. Knowing that they are stressed with sixty seconds left, two monuments to find and three photographs remaining is a different kind of knowledge, because now you can say something useful.
The experience is a monument hunt on a virtual island. You start at a campsite with five minutes, eight photographs, and two monuments to find. You take a shot by raising your face and pressing a controller button. A companion helps you navigate, tells you how many shots you have left, asks whether you want to keep the one you just took, and gives time warnings at the halfway point, at sixty seconds and at ten seconds, which are deliberately designed to raise your stress. When it detects that this has worked, it changes tack and offers something soothing instead.
Under the hood it is three parts. A biosignal processor pulls EEG, EMG, EDA and PPG off the Galea and streams it into a Python pipeline over Lab Streaming Layer. An emotion predictor turns the cleaned features into one of four states: excited, stressed, bored, or serene. An empathic adaptor takes that state, combines it with the situation, and picks both the colour treatment and the line of dialogue.
The colour mapping is not arbitrary, which I appreciated. It comes from Itten's colour system and Russell's model of affect, so yellow, red, blue and green are chosen positions in a space rather than four colours somebody liked. The voice tones were tuned after pilot feedback using Microsoft's Speech SDK.
At the end you pick your favourite photograph and it gets printed as a physical postcard you take away. A souvenir from a place that does not exist, produced by a system that was watching your nervous system while you chose it.
Being honest about what these are
They are demos. Neither paper contains an evaluation of whether any of the adaptation helped. SensoryScape's own future work section says, in effect, that the next step is to collect the physiological responses needed to improve the emotion recognition and to actually study the impact on engagement and flow.
So this is the second entry in this series that argues rather than proves, and I should not pretend otherwise.
But the shift underneath is real, and I did not fully appreciate it at the time. The assembly study running in parallel was busy demonstrating that putting a number in front of a person accomplishes very little. These demos never asked anyone to read anything. The interpretation happens inside the system, and what reaches the user is a changed world.
That is the direction that survived. Every adaptive system I have worked on since sits on this side of the line.
What stays unresolved is resolution and taste. Four emotion labels and a colour tint is a coarse vocabulary for something as textured as how a person feels. And a system that detects your stress and responds with a soothing line has to get the tone exactly right, or it reads as a machine telling you to calm down. The recognition problem was the one we were solving. The harder one is what to say once you are sure.
The papers are SensoryScape: Context-Aware Empathic VR Photography, SIGGRAPH Asia XR 2023, and VRdoGraphy: An Empathic VR Photography Experience, IEEE VR 2023, both led by Kunal Gupta with Yuewei Zhang, Tamil Selvan Gunasekaran, Nanditha Krishna, Yun Suen Pai, Philip Pitts, Conor Russomanno and Mark Billinghurst.
Involuntary Signals: 165 Hours of Feeling Sick