Machine learning had been pointed at VR motion sickness for years, and every model was trained on under two hours of video. The fix was not a better model. It was patching 100 commercial VR games so they would tell us what they were rendering.
Up to 40% of people get motion sick in VR. That number has been stable for years, and by 2023 there were plenty of machine learning papers promising to predict it from the content itself, so a store could warn you before you bought a game.
Almost all of them said the same thing in their limitations section: the training data is too small. The largest public dataset at the time held under two hours of video. Most were passive 360 clips or a single purpose-built roller coaster. A model trained on that will happily predict sickness for roller coasters and fall apart on a shooter.
So the bottleneck was not the model. It was that nobody had the data, and there was a good reason nobody had it.
Why the dataset did not exist
To know why a particular frame makes someone queasy, you need to know what the frame is doing. Camera velocity, acceleration, whether the rotation is on one axis or three. Field of view. Depth of field. Per-pixel motion vectors. How many objects are on screen.
None of that is in the recorded video. It lives inside the game engine, and for a commercial VR game you do not have the source code.
The workarounds all cost you something. Hand-labeling is accurate and does not scale past a couple of hours. Estimating motion and depth with computer vision scales but introduces noise into the labels, which is a bad trade when the labels are the whole product. Building your own VR environment gives you perfect ground truth for something nobody actually plays.
Patching the games instead
The way through, and this was Elliott Wen's work, was to stop trying to infer what the engine knew and just ask it.
The tool is called OculusHook. Give it a game's APK and it pulls out the engine library, libUnity.so or libUnreal.so, and inserts trampolines into core engine functions. When the game calls GetCamera, execution detours into our logging code, records the view and projection matrices, and then carries on into the original function. Repackage, and you have a build of a shipped commercial game that narrates its own rendering pipeline while you play it.
No source code needed. No process injection either, which Android forbids outright, so the patching is static and happens before install.
Unity and Unreal between them account for 1,403 of the 1,410 games we catalogued on the Quest store, so covering two engines covers effectively the whole platform. Out of 24 labels per frame, 23 come out automatically. Only the self-report needs a human.
The unglamorous half
The engineering was the interesting part. The reason the dataset took so long was everything around it, and that half was mine.
An earlier paper of ours had built a smaller version of the capture tool and shipped a demo dataset with it. Turning that into something the field could actually use was a different kind of problem: assemble the team, get Meta Reality Labs involved as collaborators, and then run the thing. I put the group together, drove the execution, and ran the studies.
We pulled all 1,410 games off the Meta Quest store and found the store categories were close to useless for our purposes, because developers use them to describe price, scenery and features rather than genre. So three coders went through the catalogue by hand using the Lucas and Sherry classification, we spot-checked 5%, and only then could we sample ten games from each of ten genres.
Then 500 participants, drawn from three universities on three continents, on Quest Pros, roughly 15 minutes per game, every game played by at least five different people.
The measurement choice mattered more than I expected. Most sickness research uses the Simulator Sickness Questionnaire, which you fill in after the session. That gives you one number for fifteen minutes of gameplay, and you cannot line it up with any particular frame. We used the Fast Motion Sickness scale instead: participants say a number out loud during play, nudged every 30 seconds, recorded through the headset mic and transcribed with Vosk. The result is a sickness signal with timestamps, which is the only form that is any use as a training label.
Does it work
The paper ends with two proof-of-concept models rather than a state-of-the-art claim, which I think is the right call for a dataset paper.
- Risk factor detection. A SlowFast model reading one second of video predicts whether the camera is moving too fast, accelerating too hard, or rotating on multiple axes, at 0.71 to 0.82 accuracy depending on the factor. It holds up when you split by user and when you split by content, which is the generalisation the earlier work could not show.
- Object density estimation. A VideoMAE model sorts Beat Saber footage into low, medium and high on-screen object counts, as a stand-in for scene complexity.
Neither is the point. The point is that both were trained in an afternoon on top of a substrate that did not exist the year before.
What I took from it
It won a Best Paper award at IEEE VR 2024, in the journal track, and it is now the most cited thing I have been part of from that period. My reading is that it won because the field was blocked on data rather than on ideas, and the paper unblocked it. That is not the kind of contribution I was trained to aim for. I spent my PhD building systems that demonstrated something. This one built a floor for other people to stand on, and the citations suggest that was worth more.
Worth stating the limits plainly. Physiological signals were only captured for about 10% of participants, since most headsets have no sensor for it. It is all lab-collected on one headset. And detecting that a frame is risky still says nothing about what a system should do in response, which is the question the rest of this series keeps running into.
The paper is VR.net: A Real-world Dataset for Virtual Reality Motion Sickness Research, IEEE Transactions on Visualization and Computer Graphics 30(5), 2024, by Elliott Wen, Chitralekha Gupta, Prasanth Sasikumar, Mark Billinghurst, James Wilmott, Emily Skow, Arindam Dey and Suranga Nanayakkara. The dataset is at vrnet.ahlab.org and the patching tool is at vrpatcher.ahlab.org.
Involuntary Signals: Nobody Looked at the Heart Rate