• Enabling Emotional Intelligence with VR and Physiological Sensing (2024 +)
• Guiding vs. Restoring Attention in Virtual Environments (2019 – 2023)
• Enhancing Collaborative Communication and Dataset Exploration in VR (2019-2024)
• Affective Tasks for Eliciting Emotion in VR (2017+)
• Real-Time Emotion and Empathy Reporting in VR (2023+)
• Gaze-Driven User Identification in VR
• Recognizing Emotions from Physiological Responses
• Detecting and Responding to Student Distraction with Pedagogical Agents (2018+)

---

Enabling Emotional Intelligence with VR and Physiological Sensing (2024 +)

Researchers: Niloofar Heidarikohol

Stress can noticeably reduce students’ well-being and academic performance, and becoming aware of one’s emotions and physiological state, for example by regulating breathing, helps people manage stressful moments. This project explores how VR with biofeedback and physiological sensing can help young people build emotional intelligence (EI), emotion self-regulation, and emotional resilience.

We first reviewed how EI concepts are applied in immersive and non-immersive technologies (40 of 822 papers, following PRISMA) and how immersive technologies have been used for empathic computing from 2000 to 2024. Building on these reviews, we designed a nature-based VR environment that measures heart rate, skin conductivity, and pupil size and movement, and presents the user’s heart rate in real time as a visual pulse, a heartbeat sound, or a vibration through a haptic vest. We are using data from user experiments to refine the environment and compare these feedback modalities.

Application: VR Nature Scene with Heart Rate Biofeedback

Publications:

N. Heidarikohol and C. W. Borst, “A Survey of Emotional Intelligence Applications in Immersive and Non-Immersive Technologies” 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. 2025, pp. 997-1004, doi: 10.1109/VRW66409.2025.00202. (PDF)

U. A. Jinan, N. Heidarikohol, C. W. Borst, M. Billinghurst, S. Jung, “A systematic review of using immersive technologies for empathic computing from 2000-2024.” Empathic Computing 1.1 (2025): 202501-202501. (PDF)

---

Guiding vs. Restoring Attention in Virtual Environments (2019 – 2023)

Researchers: Andrew Yoshimura, Nicholas Lipari, and Jason Woodworth

In VR training, a short distraction such as a notification can pull a user’s attention away from a target. Visual cues are usually evaluated on “search and selection” tasks that guide users to unknown targets, so we compared 9 eye-tracked cues on both a guidance task and a new restoration task, in which users must return their gaze to an object after a distraction. The study varied object spacing, gaze dwell time, and distraction distance and duration.

Cues that directly connect the user’s gaze to the target generally worked better than cues that only suggest a direction, and cue effectiveness followed different patterns for restoration than for guidance, likely because users remember where the object was. For restoration, some cues added complexity that reduced rather than improved performance, so results from guidance studies do not carry over directly to restoring attention.

Application: Offshore Oil Rig Training with a Gaze-Aware Teacher

Videos:

Cues and procedures for restoring attention:

Gaze-triggered visual cues to restore attention (IEEE VR poster):

 

Related Publications:

J. W. Woodworth and C. W. Borst, “Visual cues in VR for guiding attention vs. restoring attention after a short distraction,” Computers & Graphics (118), pp. 194-209, 2024, doi: 10.1016/j.cag.2023.12.008. (PDF)

J. W. Woodworth, A. Yoshimura, N. G. Lipari and C. W. Borst, “Design and Evaluation of Visual Cues for Restoring and Guiding Visual Attention in Eye-Tracked VR,” 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Shanghai, China, 2023, pp. 442-450, doi: 10.1109/VRW58643.2023.00096. (link)

---

Enhancing Collaborative Communication and Dataset Exploration in VR (2019-2024)

Researchers: Adil Khokhar, Md Istiak Jaman Ami, Niloofar Heidarikohol

In a remote one-on-one VR lesson, a tutor usually cannot see where the student is looking. When a student asks about “this ridge” in a 3D terrain dataset, the tutor has to ask follow-up questions to work out which one they mean. We study how showing the student’s gaze to the tutor changes this kind of spatial guidance, using a collaborative geosciences environment with terrain of the Yucatán Peninsula and Lake Constance and a live cursor at the student’s combined eye and head gaze point.

Tutors guided a confederate student through locating, aligning, and matching terrain features, with and without the gaze cursor. With gaze visualization, tutors corrected the student in less time and with fewer words, suggesting a more efficient, visually oriented way of giving spatial guidance.

Application: Remote Tutoring on Terrain Datasets with Live Gaze

Videos:

 

Publications:

Khokhar, A., Borst, C. W., Ami, M. I. J., & Heidarikohol, N. (2025). Enhancing Collaborative Dataset Exploration in VR: The Impact of Gaze Visualization on Tutors’ Spatial Guidance and Communication. 596–603. https://doi.org/10.1109/VRW66409.2025.00125
Borst, C. W., Lipari, N. G., & Woodworth, J. W. (2018). Teacher-Guided Educational VR: Assessment of Live and Prerecorded Teachers Guiding Virtual Field Trips. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 467–474. https://doi.org/10.1109/VR.2018.8448286

Woodworth, J. W., Broussard, D., & Borst, C. W. (2019). Collaborative Interaction in Large Explorative Environments. Symposium on Spatial User Interaction, 1–2. https://doi.org/10.1145/3357251.3360017

---

Affective Tasks for Eliciting Emotion in VR (2017+)

Researchers: Jason Woodworth

Achieving automated emotion recognition requires large datasets of users’ physiological and behavioral responses to emotional stimuli. To help capture such datasets, researchers have created libraries of stimuli ranging from images to videos to virtual environments. However, many of these libraries contain only passive stimuli that don’t require the user to actively interact with them. Those that contain video often use narratives that expect a user to empathize with characters in ways that may not be culturally ubiquitous. The emotions they target also tend to not relate to those associated with the training or educational tasks common in VR, such as frustration or boredom.

The capability of VR to enhance engagement through its immersive interactivity is a core selling point of the medium, and so we sought to create a library of active affective stimuli that would use VR to quickly elicit emotions from the user without need for narrative devices. Our initial library includes 4 affective tasks (and alternate neutral variants) that aim to elicit frustration, confusion, excitement, and boredom. Our study showed general success in eliciting a broad range of emotions on par with those in a passive video library. Future work looks to extend this set of tasks with others and make them more universally usable.

Application: Affective Task Library for VR

Videos:

Presentation of early work describing each task.

Video description of the experiment presented at IEEE 2024

 

Related Publications:

J. W. Woodworth and C. W. Borst, “Design and Validation of a Library of Active Affective Tasks for Emotion Elicitation in VR,” 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Orlando, FL, USA, 2024, pp. 398-407, doi: 10.1109/VR58804.2024.00061. (PDF)

J. W. Woodworth and C. W. Borst, “Designing immersive affective tasks for emotion elicitation in virtual reality,” 2nd Momentary Emotion Elicitation and Capture Workshop, 2021(PDF)

 

 

---

Real-Time Emotion and Empathy Reporting in VR (2023+)

Researchers: Jason Woodworth, Md Istiak Jaman Ami

Emotional responses change throughout an experience such as a 360° video or VR task, but most interfaces for reporting emotion are used only after the experience, which reduces it to a single data point. We design and evaluate interfaces that let users report their emotions continuously while immersed.

A preliminary study compared five interfaces for reporting valence and arousal, and a primary study compared four refined designs for precision and user preference. A face icon responding to valence and arousal and a gradient color wheel were intuitive, precise, and unobtrusive, and the choice of interface significantly affected the ratings given.

We then extended this work to two co-occurring empathic responses, empathic concern (EC) and personal distress (PD). Viewers of a VR180 video of a crying infant reported both at once with two rotary knobs, each mapped to its own visual indicator. In a study with 40 participants, both ratings rose more during higher-intensity crying, with PD following the stimulus slightly more closely. Participants coordinated the two channels in different ways, from using only one, to switching one way, to alternating back and forth. They reported low effort and little disruption to viewing, which shows that continuous two-channel self-report is feasible in VR.

Video:

Related Publications:

M. I. J. Ami, J. W. Woodworth, H.-C. Lin and C. W. Borst, “Reporting Two Feelings at Once: An Empirical Characterization of Dual-Channel Continuous Self-Report in Immersive VR,” 32nd ACM Symposium on Virtual Reality Software and Technology (VRST ’26), Sendai, Japan, 2026, doi: 10.1145/3822517.3848737. (to appear)

J. W. Woodworth and C. W. Borst, “Study of Interfaces for Time-Continuous Emotion Reporting and the Relationship Between Interface and Reported Emotion,” 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bellevue, WA, USA, 2024, pp. 884-892, doi: 10.1109/ISMAR62088.2024.00104. (PDF)

M. I. J. Ami, J. W. Woodworth and C. W. Borst, “Design of Time-Continuous Rating Interfaces for Collecting Empathic Responses in VR, and Initial Evaluation with VR180 Video Viewing,” 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2024, pp. 1041-1042, doi: 10.1109/VRW62533.2024.00317. (PDF)

J. W. Woodworth and C. W. Borst, “Design of Time-Continuous Emotion Rating Interfaces,” Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (VRST ’23), 2023, doi: 10.1145/3611659.3617199. (PDF)

 

---

Gaze-Driven User Identification in VR (2022 – 2024)

Researchers: Bhoj Karki

VR headsets with eye tracking and physiological sensors make it possible to recognize who is using them, which supports personalization and adaptive training. It is unclear, however, what drives identification across different tasks: the emotional content of a task, its interaction structure, or the user’s gaze behavior.

We collected eye-tracking and physiological data from 33 participants, each performing four training-relevant affective VR tasks and four structurally matched neutral versions, and trained machine learning and deep learning models to identify users across tasks. Identification accuracy was largely preserved when the affective elements were removed, as long as task structure and gaze behavior stayed consistent. Eye-tracking features gave more stable user signatures than physiological signals, which added complementary gains when training spanned several tasks.

Video: see Affective Tasks for Eliciting Emotion in VR for a video of the tasks.

Related Publication:

B. Karki, B. KC, A. A. Khokhar and C. W. Borst, “Gaze-Driven User Identification in Affective VR Tasks,” 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026, pp. 120-127, doi: 10.1109/VRW70859.2026.00027. (link)

 

---

Recognizing Emotions from Physiological Responses (2022-2024)

Researchers: Bharat KC

Emotion recognition models are usually trained on physiological responses to passive stimuli such as videos, although interactive VR tasks can elicit stronger emotions. We collected a multimodal dataset of EEG, electrodermal activity, heart rate, and pupil response from users experiencing both passive stimuli (360° VR videos) and active stimuli (interactive VR tasks).

We designed emotion recognition classifiers that combine transformer and fully convolutional network architectures. Models trained on the interactive VR tasks performed best, reaching about 97% accuracy for valence-arousal classification, and EEG contributed most to performance. Generalizing across passive and active stimuli remains a challenge.

Video: see Affective Tasks for Eliciting Emotion in VR for a video of the tasks.

Related article:

Bharat, K. C., Karki, B., Khokhar, A., Woodworth, J. W., & Borst, C. W. (2025, March). Emotion Recognition in Interactive VR Tasks and in 360VR Videos with Transformer-Based Approach and Multimodal Sensing. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 299-306). IEEE.(PDF)

 

---

Detecting and Responding to Student Distraction with Pedagogical Agents (2018+)

Researchers: Adil Khokhar

Distractions can interrupt critical learning moments in educational VR. We detect a student’s distraction from the head, hand controller, and eye motion that a headset already records: trained on data from an educational field trip to a virtual offshore oil rig with in-game distraction prompts, a deep CNN-LSTM classifier detected moments of distraction with over 94% accuracy.

To respond to distraction, our pedagogical agent, a teacher avatar built from prerecorded Kinect clips, adapts its guidance to the student’s eye gaze in real time. Generalized hotspots, higher-level sensors built on eye-tracking attention metrics, let it respond to kinds of gaze behavior rather than single trigger objects. In an oil rig field trip study comparing Continue, Pause, and Respond behaviors under two levels of distraction, participants rated the more interactive behaviors as more appropriate, more natural, and less strange than the default behavior.

Application: Offshore Oil Rig Training with a Gaze-Aware Teacher

Videos:

A pedagogical agent that responds to eye tracking:

The oil rig study, with distraction prompts on the in-game phone and participant ratings:

Publications:

Khokhar, A., & Borst, C. W. (2022). Towards Improving Educational Virtual Reality by Classifying Distraction using Deep Learning. ICAT-EGVE 2022, The Eurographics Association. doi: 10.2312/egve.20221279. (PDF)

Khokhar, A., & Borst, C. W. (2022). Modifying Pedagogical Agent Spatial Guidance Sequences to Respond to Eye-Tracked Student Gaze in VR. Symposium on Spatial User Interaction (SUI ’22), Article 12. (link)

Khokhar, A., Yoshimura, A., & Borst, C. W. (2020). Modified Playback of Avatar Clip Sequences Based on Student Attention in Educational VR. 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 850–851. (link)

Khokhar, A., Yoshimura, A., & Borst, C. W. (2019). Pedagogical Agent Responsive to Eye Tracking in Educational VR. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1018–1019. (link)