2026
A Statistical Abstraction Framework for Integrating Heterogeneous VR Datasets in Ordinal Cybersickness Prediction
Jyotirmay Nag Setu, John Quarles
IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2026 Conference
We developed a comprehensive statistical feature abstraction framework combined with dual-head ordinal regression to enable cybersickness prediction across heterogeneous VR datasets. Our approach transforms diverse sensor modalities from VR.net (Meta Quest Pro proprietary systems), SIM21, and VRWalking (HTC VIVE/Tobii) platforms into standardized statistical descriptors that capture cybersickness-relevant behavioral patterns while remaining invariant to study design and data collection methodologies.
A Statistical Abstraction Framework for Integrating Heterogeneous VR Datasets in Ordinal Cybersickness Prediction
Jyotirmay Nag Setu, John Quarles
IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 2026 Conference
We developed a comprehensive statistical feature abstraction framework combined with dual-head ordinal regression to enable cybersickness prediction across heterogeneous VR datasets. Our approach transforms diverse sensor modalities from VR.net (Meta Quest Pro proprietary systems), SIM21, and VRWalking (HTC VIVE/Tobii) platforms into standardized statistical descriptors that capture cybersickness-relevant behavioral patterns while remaining invariant to study design and data collection methodologies.
2025
PatchFusionVR: Multitask prediction of user gaze, reaction time, and cognitive load in virtual reality from multimodal signals
Md Irfan Pavel, M Rasel Mahmud, Jyotirmay Nag Setu, Kevin Desai, John Quarles
ACM Symposium on Virtual Reality Software and Technology 2025 Conference
We developed a unified multitask model based on the MultiPatchFormer architecture, which processes multimodal VR signals through dual patch projection branches for gaze and classification inputs. The model employs multiscale patch embeddings, cross-attention between gaze and classification pathways, channel attention, and transformer encoders to jointly predict continuous user gaze and classify reaction time, cognitive load (mental load and physical load).
PatchFusionVR: Multitask prediction of user gaze, reaction time, and cognitive load in virtual reality from multimodal signals
Md Irfan Pavel, M Rasel Mahmud, Jyotirmay Nag Setu, Kevin Desai, John Quarles
ACM Symposium on Virtual Reality Software and Technology 2025 Conference
We developed a unified multitask model based on the MultiPatchFormer architecture, which processes multimodal VR signals through dual patch projection branches for gaze and classification inputs. The model employs multiscale patch embeddings, cross-attention between gaze and classification pathways, channel attention, and transformer encoders to jointly predict continuous user gaze and classify reaction time, cognitive load (mental load and physical load).
Towards Cybersickness Severity Classification from VR Gameplay Videos Using Transfer Learning and Temporal Modeling
Jyotirmay Nag Setu, Kevin Desai, John Quarles
IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025 Workshop
In this study, we utilize transfer learning to extract high-level visual features from VR gameplay videos using the InceptionV3 model pre-trained on the ImageNet dataset. These features are then passed to a Long Short-Term Memory (LSTM) network to capture the temporal dynamics of the VR experience and predict cybersickness severity over time.
Towards Cybersickness Severity Classification from VR Gameplay Videos Using Transfer Learning and Temporal Modeling
Jyotirmay Nag Setu, Kevin Desai, John Quarles
IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2025 Workshop
In this study, we utilize transfer learning to extract high-level visual features from VR gameplay videos using the InceptionV3 model pre-trained on the ImageNet dataset. These features are then passed to a Long Short-Term Memory (LSTM) network to capture the temporal dynamics of the VR experience and predict cybersickness severity over time.
Predicting and Explaining Cognitive Load, Attention, and Working Memory in Virtual Multitasking
Jyotirmay Nag Setu, Joshua M Le, Ripan Kumar Kundu, Barry Giesbrecht, Tobias Höllerer, Khaza Anuarul Hoque, Kevin Desai, John Quarles
IEEE transactions on visualization and computer graphics 2025 Journal
This study shows the similarity between cybersickness and cognitive load prediciton research. We showed how the same modalities can be used to build prediciton models for cogntiive load assessment.
Predicting and Explaining Cognitive Load, Attention, and Working Memory in Virtual Multitasking
Jyotirmay Nag Setu, Joshua M Le, Ripan Kumar Kundu, Barry Giesbrecht, Tobias Höllerer, Khaza Anuarul Hoque, Kevin Desai, John Quarles
IEEE transactions on visualization and computer graphics 2025 Journal
This study shows the similarity between cybersickness and cognitive load prediciton research. We showed how the same modalities can be used to build prediciton models for cogntiive load assessment.
Investigating the potential of multi-task learning for enhancing ssq prediction
Anjan Kumar Dev, Purnata Saha, Jyotirmay Nag Setu, Samuel Ang, John Quarles, Maryam Tabar
IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025 Workshop
Deep learning is widely used to forecast cybersickness to inform mitigation efforts. However, prior work mostly focused on predicting the Fast Motion Sickness score, which does not provide detailed information on cybersickness symptoms. To fill this gap, this paper focuses on forecasting cybersickness symptoms and evaluates the effectiveness of multi-task learning. The results show that multitask learning does not always outperform the traditional single-task learning approach in this domain.
Investigating the potential of multi-task learning for enhancing ssq prediction
Anjan Kumar Dev, Purnata Saha, Jyotirmay Nag Setu, Samuel Ang, John Quarles, Maryam Tabar
IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) 2025 Workshop
Deep learning is widely used to forecast cybersickness to inform mitigation efforts. However, prior work mostly focused on predicting the Fast Motion Sickness score, which does not provide detailed information on cybersickness symptoms. To fill this gap, this paper focuses on forecasting cybersickness symptoms and evaluates the effectiveness of multi-task learning. The results show that multitask learning does not always outperform the traditional single-task learning approach in this domain.
2024
Mazed and confused: A dataset of cybersickness, working memory, mental load, physical load, and attention during a real walking task in VR
Jyotirmay Nag Setu, Joshua M Le, Ripan Kumar Kundu, Barry Giesbrecht, Tobias Höllerer, Khaza Anuarul Hoque, Kevin Desai, John Quarles
IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024 Conference
A dataset that collects multimodal data including eye-tracking, head-tracking, Hear Rate (HR), Galvanic Skin Response (GSR) alongside ground truth values for cybersickness, physical load, mental load, attention and working memory while participants are performing multiple tasks.
Mazed and confused: A dataset of cybersickness, working memory, mental load, physical load, and attention during a real walking task in VR
Jyotirmay Nag Setu, Joshua M Le, Ripan Kumar Kundu, Barry Giesbrecht, Tobias Höllerer, Khaza Anuarul Hoque, Kevin Desai, John Quarles
IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2024 Conference
A dataset that collects multimodal data including eye-tracking, head-tracking, Hear Rate (HR), Galvanic Skin Response (GSR) alongside ground truth values for cybersickness, physical load, mental load, attention and working memory while participants are performing multiple tasks.