• Enabling Emotional Intelligence with VR and Physiological Sensing (2024 +)
• Guiding vs. Restoring Attention in Virtual Environments (2019 – 2023)
• Enhancing Collaborative Communication and Dataset Exploration in VR (2019-2024)
• Affective Tasks for Eliciting Emotion in VR (2017+)
• Real-Time Emotion and Empathy Reporting in VR (2023+)
• Gaze-Driven User Identification in VR
• Recognizing Emotions from Physiological Responses
• Detecting and Responding to Student Distraction with Pedagogical Agents (2018+)

---

Enabling Emotional Intelligence with VR and Physiological Sensing (2024 +)

Researchers: Niloofar Heidarikohol

Description:

Stress is a state of worry and mental pressure that people experience when facing difficult circumstances. According to a Gallup poll, 66% of students experience stress and 51% experience worry. Stress affects people cognitively, behaviorally, emotionally, and physically, and it can noticeably reduce productivity in daily tasks such as academic work. Techniques such as regulating breathing and becoming aware of the body’s responses can help during peaks of negative emotion. This project explores how a VR environment equipped with biofeedback and physiological sensors can help young people build emotional intelligence (EI), emotion self-regulation, and emotional resilience.

The first phase of the project produced two survey papers. The first, published at IEEE VRW 2025, reviews how EI concepts are applied in immersive and non-immersive technologies; using the PRISMA method, we narrowed 822 papers down to 40 for analysis. The second is a systematic review of immersive technologies for empathic computing from 2000 to 2024. Building on these reviews, we designed a nature-based VR environment for developing EI and awareness of one’s physiological state. Sensors measure heart rate, skin conductivity, and pupil size and movement, and the system converts heart rate data into real-time feedback presented visually, through sound, or through a haptic vest.

We collected data from user experiments to refine the environment and the heart rate feedback modalities. Insights from these initial designs and experiments will guide the next stage of the project. The University of Louisiana at Lafayette has funded the project for two consecutive years, starting in August 2024.

Application: VR Nature Scene with Heart Rate Biofeedback

Publications:

N. Heidarikohol and C. W. Borst, “A Survey of Emotional Intelligence Applications in Immersive and Non-Immersive Technologies” 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France. 2025, pp. 997-1004, doi: 10.1109/VRW66409.2025.00202. (PDF)

U. A. Jinan, N. Heidarikohol, C. W. Borst, M. Billinghurst, S. Jung, “A systematic review of using immersive technologies for empathic computing from 2000-2024.” Empathic Computing 1.1 (2025): 202501-202501. (PDF)

---

Guiding vs. Restoring Attention in Virtual Environments (2019 – 2023)

Researchers: Andrew Yoshimura, Nicholas Lipari, and Jason Woodworth

Description:

Distraction in VR training environments may be mitigated with a visual cue intended to guide user attention to a target. A survey of related literature suggests a past focus on “search and selection” tasks to evaluate a cue’s capability for guidance to unknown targets, but they do not study the effect of guiding a user’s attention to a known target after it is lost. To address this, we are investigating visual attention cues in VR with a focus on how to restore attention attention when a short distraction (e.g., a notification) shifts focus away from a target.

We performed a study comparing 9 eye-tracked cues with a new type of task different from the standard “search and selection.” We initially conducted a tuning study to find ideal parameters for each cue, including settings like cue shape and visual fade with gaze angle from target. Our main study then included a guidance task, in which subjects gazed at new targets in a randomized order, and a restoration task, in which gaze sequences were interrupted by distraction events after which gaze must be returned to the target object. We considered a wider variety of factors and metrics than previous studies, varying object spacing, gaze dwell time, and distraction distance and duration, resulting in a 10x3x2x2 within-subjects experiment design.

Results show a general positive trend for cues that directly connect the user’s gaze to the target rather than indirectly suggesting direction. Results further reveal different patterns of cue effectiveness for the restoration task than for conventional guidance. This may be attributed to knowledge that subjects have about the location of the object from which they were distracted. An implication for more complex dis traction tasks is that we expect them to be between the short distraction and regular guidance in terms of memory of object position. So, we speculate cue performance for other tasks would vary between the short distraction and guidance results. For restoration, some cues add complexity that reduces, rather than improves, performance. Future studies may further investigate this difference.

Application: Offshore Oil Rig Training with a Gaze-Aware Teacher

Videos:

 

Related Publications:

J. W. Woodworth and C. W. Borst, “Visual cues in VR for guiding attention vs. restoring attention after a short distraction,” Computers & Graphics (118), pp. 194-209, 2024, doi: 10.1016/j.cag.2023.12.008. (PDF)

J. W. Woodworth, A. Yoshimura, N. G. Lipari and C. W. Borst, “Design and Evaluation of Visual Cues for Restoring and Guiding Visual Attention in Eye-Tracked VR,” 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Shanghai, China, 2023, pp. 442-450, doi: 10.1109/VRW58643.2023.00096. (link)

---

Enhancing Collaborative Communication and Dataset Exploration in VR (2019-2024)

Researchers: Adil Khokhar, Md Istiak Jaman Ami, Niloofar Heidarikohol

Description:

Effective collaboration in VR depends on shared visuals and also on the nonverbal cues people use to understand each other. Eye gaze is one of the most important of these cues: it reveals attention, intent, and even emotional state, and tutors and learners use it to align their understanding. In a remote one-on-one VR lesson, however, a tutor usually cannot see where the student is looking. When a student asks about “this ridge” in a 3D terrain dataset, the tutor has to ask follow-up questions to work out which ridge they mean, which interrupts the lesson. We are investigating how showing a student’s gaze to the tutor affects this kind of spatial guidance.

Our collaborative geosciences VR environment displays high-resolution terrain of the Yucatán Peninsula and Lake Constance and can show several datasets at once as overlays or tiles. A live cursor shows the tutor the student’s combined eye and head gaze point, so the tutor can identify what the student is referring to without extra verbal clarification. In our study, subjects acted as tutors guiding a student through three terrain tasks: locating, aligning, and matching geological features. The student was a confederate, blind to the condition, who controlled when mistakes happened and when to ask for guidance. Each tutor performed the tasks with and without live gaze visualization, and we analyzed subjective ratings, task completion times, the number of words spoken, and time spent looking at the student.

With gaze visualization, tutors needed less time and fewer verbal instructions to correct the student. It also changed how they guided: they used fewer words and finished tasks faster, which suggests a more efficient, visually oriented communication strategy. Overall, gaze visualization helps tutors in educational VR give spatial guidance with less reliance on long verbal descriptions, which benefits tasks that demand precise spatial orientation.

Application: Remote Tutoring on Terrain Datasets with Live Gaze

Videos:

 

Publications:

Khokhar, A., Borst, C. W., Ami, M. I. J., & Heidarikohol, N. (2025). Enhancing Collaborative Dataset Exploration in VR: The Impact of Gaze Visualization on Tutors’ Spatial Guidance and Communication. 596–603. https://doi.org/10.1109/VRW66409.2025.00125
Borst, C. W., Lipari, N. G., & Woodworth, J. W. (2018). Teacher-Guided Educational VR: Assessment of Live and Prerecorded Teachers Guiding Virtual Field Trips. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 467–474. https://doi.org/10.1109/VR.2018.8448286

Woodworth, J. W., Broussard, D., & Borst, C. W. (2019). Collaborative Interaction in Large Explorative Environments. Symposium on Spatial User Interaction, 1–2. https://doi.org/10.1145/3357251.3360017

---

Affective Tasks for Eliciting Emotion in VR (2017+)

Researchers: Jason Woodworth

Achieving automated emotion recognition requires large datasets of users’ physiological and behavioral responses to emotional stimuli. To help capture such datasets, researchers have created libraries of stimuli ranging from images to videos to virtual environments. However, many of these libraries contain only passive stimuli that don’t require the user to actively interact with them. Those that contain video often use narratives that expect a user to empathize with characters in ways that may not be culturally ubiquitous. The emotions they target also tend to not relate to those associated with the training or educational tasks common in VR, such as frustration or boredom.

The capability of VR to enhance engagement through its immersive interactivity is a core selling point of the medium, and so we sought to create a library of active affective stimuli that would use VR to quickly elicit emotions from the user without need for narrative devices. Our initial library includes 4 affective tasks (and alternate neutral variants) that aim to elicit frustration, confusion, excitement, and boredom. Our study showed general success in eliciting a broad range of emotions on par with those in a passive video library. Future work looks to extend this set of tasks with others and make them more universally usable.

Videos:

Presentation of early work describing each task.

Video description of the experiment presented at IEEE 2024

 

Related Publications:

J. W. Woodworth and C. W. Borst, “Design and Validation of a Library of Active Affective Tasks for Emotion Elicitation in VR,” 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Orlando, FL, USA, 2024, pp. 398-407, doi: 10.1109/VR58804.2024.00061. (PDF)

J. W. Woodworth and C. W. Borst, “Designing immersive affective tasks for emotion elicitation in virtual reality,” 2nd Momentary Emotion Elicitation and Capture Workshop, 2021(PDF)

 

 

---

Real-Time Emotion and Empathy Reporting in VR (2023+)

Researchers: Jason Woodworth, Md Istiak Jaman Ami

Emotional responses change throughout an experience such as a 360° video or VR task, but most interfaces for reporting emotion are used only after the experience, which reduces it to a single data point. We design and evaluate interfaces that let users report their emotions continuously while immersed.

A preliminary study compared five interfaces for reporting valence and arousal, and a primary study compared four refined designs for precision and user preference. A face icon responding to valence and arousal and a gradient color wheel were intuitive, precise, and unobtrusive, and the choice of interface significantly affected the ratings given.

We then extended this work to two co-occurring empathic responses, empathic concern (EC) and personal distress (PD). Viewers of a VR180 video of a crying infant reported both at once with two rotary knobs, each mapped to its own visual indicator. In a study with 40 participants, both ratings rose more during higher-intensity crying, with PD following the stimulus slightly more closely. Participants coordinated the two channels in different ways, from using only one, to switching one way, to alternating back and forth. They reported low effort and little disruption to viewing, which shows that continuous two-channel self-report is feasible in VR.

Video:

Related Publications:

M. I. J. Ami, J. W. Woodworth, H.-C. Lin and C. W. Borst, “Reporting Two Feelings at Once: An Empirical Characterization of Dual-Channel Continuous Self-Report in Immersive VR,” 32nd ACM Symposium on Virtual Reality Software and Technology (VRST ’26), Sendai, Japan, 2026, doi: 10.1145/3822517.3848737. (to appear)

J. W. Woodworth and C. W. Borst, “Study of Interfaces for Time-Continuous Emotion Reporting and the Relationship Between Interface and Reported Emotion,” 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bellevue, WA, USA, 2024, pp. 884-892, doi: 10.1109/ISMAR62088.2024.00104. (PDF)

M. I. J. Ami, J. W. Woodworth and C. W. Borst, “Design of Time-Continuous Rating Interfaces for Collecting Empathic Responses in VR, and Initial Evaluation with VR180 Video Viewing,” 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2024, pp. 1041-1042, doi: 10.1109/VRW62533.2024.00317. (PDF)

J. W. Woodworth and C. W. Borst, “Design of Time-Continuous Emotion Rating Interfaces,” Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (VRST ’23), 2023, doi: 10.1145/3611659.3617199. (PDF)

 

---

Gaze-Driven User Identification in VR (2022 – 2024)

Researchers: Bhoj Karki

VR headsets with eye tracking and physiological sensors make it possible to recognize who is using them, which supports personalization and adaptive training. It is unclear, however, what drives identification across different tasks: the emotional content of a task, its interaction structure, or the user’s gaze behavior.

We collected eye-tracking and physiological data from 33 participants, each performing four training-relevant affective VR tasks and four structurally matched neutral versions, and trained machine learning and deep learning models to identify users across tasks. Identification accuracy was largely preserved when the affective elements were removed, as long as task structure and gaze behavior stayed consistent. Eye-tracking features gave more stable user signatures than physiological signals, which added complementary gains when training spanned several tasks.

Related Publication:

B. Karki, B. KC, A. A. Khokhar and C. W. Borst, “Gaze-Driven User Identification in Affective VR Tasks,” 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026, pp. 120-127, doi: 10.1109/VRW70859.2026.00027. (link)

 

---

Recognizing Emotions from Physiological Responses (2022-2024)

Researchers: Bharat KC

Emotion recognition models are usually trained on physiological responses to passive stimuli such as videos, although interactive VR tasks can elicit stronger emotions. We collected a multimodal dataset of EEG, electrodermal activity, heart rate, and pupil response from users experiencing both passive stimuli (360° VR videos) and active stimuli (interactive VR tasks).

We designed emotion recognition classifiers that combine transformer and fully convolutional network architectures. Models trained on the interactive VR tasks performed best, reaching about 97% accuracy for valence-arousal classification, and EEG contributed most to performance. Generalizing across passive and active stimuli remains a challenge.

Related article:

Bharat, K. C., Karki, B., Khokhar, A., Woodworth, J. W., & Borst, C. W. (2025, March). Emotion Recognition in Interactive VR Tasks and in 360VR Videos with Transformer-Based Approach and Multimodal Sensing. In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 299-306). IEEE.(PDF)

 

---

Detecting and Responding to Student Distraction with Pedagogical Agents (2018+)

Researchers: Adil Khokhar

Distractions can interrupt critical learning moments in educational VR and cause students to miss important information. We study how a VR system can detect a student’s distraction from the motion and eye data that a headset already records, and how a pedagogical agent, an automated teacher avatar, can respond to it in real time.

For detection, we created a dataset of generalized motion features from head, hand controller, and eye tracking, such as how fast the head turns, pupil diameter, and how open the eyes are. The data were recorded during an educational VR field trip to a virtual offshore oil rig, in which an in-game mobile device occasionally prompts a distraction. A deep CNN-LSTM classifier trained on this dataset detected moments of distraction with over 94% accuracy.

The agent is built from prerecorded instructor clips captured with a Kinect V2. An annotation-driven sequencer plays, pauses, seeks within, and rewinds these clips based on real-time eye tracking of the student’s gaze, guided by annotations that mark critical moments in the recordings. Attention is detected by a behavior-based AI that goes beyond hotspots that simply trigger when a user looks at an object: low-level sensors read the eye-tracking data, sensor combiners compute attention metrics, and higher-level generalized hotspots act on those metrics to select the agent’s response.

In a virtual oil rig field trip study comparing three agent behaviors, Continue, Pause, and Respond, under two levels of distraction, participants rated the more interactive behaviors as more appropriate, more natural, and less strange than the default behavior, and they perceived the minimal response (Pause) and the more dynamic response (Respond) differently.

Application: Offshore Oil Rig Training with a Gaze-Aware Teacher

Videos:

Publications:

Khokhar, A., & Borst, C. W. (2022). Towards Improving Educational Virtual Reality by Classifying Distraction using Deep Learning. ICAT-EGVE 2022, The Eurographics Association. doi: 10.2312/egve.20221279. (PDF)

Khokhar, A., & Borst, C. W. (2022). Modifying Pedagogical Agent Spatial Guidance Sequences to Respond to Eye-Tracked Student Gaze in VR. Symposium on Spatial User Interaction (SUI ’22), Article 12. (link)

Khokhar, A., Yoshimura, A., & Borst, C. W. (2020). Modified Playback of Avatar Clip Sequences Based on Student Attention in Educational VR. 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 850–851. (link)

Khokhar, A., Yoshimura, A., & Borst, C. W. (2019). Pedagogical Agent Responsive to Eye Tracking in Educational VR. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1018–1019. (link)