Project XR.
Revolutionizing VR and AR technologies
with Center for Transformational Play at Carnegie Mellon University
Learn MoreAbout
Project XR
Project XR is a VR application coupled with an AI agent designed to help individuals overcome and cope with traumas, fears, and personal issues. By leveraging AI, the app can recognize and respond to users’ emotional cues in real time, providing personalized guidance that adapts to individual needs. Additionally, with the integration of Emotion Regulation Training (ERT) and Cognitive Reappraisal Skills Training (CRST) the application is able to support users in developing healthier cognitive routines. By combining the VR and web applications the client can use the data to personalize user experiences, identify patterns gaining deeper insights into individual behaviors, and will contribute to the research in evaluating AI's strengths and limitations. The objective is to help users achieve a balanced state of mind, recognizing that changing thought patterns can significantly affect emotional well-being.
About Client
GIST VR Project
GIST VR LAB is a creative lab in Carnegie Mellon University’s Center for Transformational Play (CTP). The lab's main focus is to develop AI with virtual reality technology for well-being, which focuses on self-critical thoughts and stress management. Additionally, the lab has created advanced virtual reality and augmented reality tech that centers around immersive visualization, interactive experiences and interdisciplinary applications like robotics, medical imaging or education. In accomplishing this, GIST VR LAB has worked with various organizations such as Heinz Foundation, Google, Amazon or NSF which provides funding for different research projects.
The GIST VR LAB’s current director, Professor Jessica Hammer, is renowned for her groundbreaking work in extended reality also known as XR. Hammer has also done extensive research and work in other interdisciplinary fields such as game design, psychology, and health technologies. Hammer has been honored with prestigious awards, including the World Economic Forum Young Scientist award and the Okawa Award, recognizing her significant contributions to these fields.
Team
Deliverables
Project XR, in collaboration with the GIST VR Project at Carnegie Mellon University's Human-Computer Interaction Institute, strives to revolutionize mental health solutions through Extended Reality (XR) technologies. With a focus on combating self-critical and negative thoughts, the program pioneers tangible experiences that foster positive impact. By creating an immersive platform, the collaboration aims to provide users with innovative solutions, utilizing XR to enhance self-awareness and promote mental well-being. Let’s take a closer look at the incredible teams driving innovation in Project XR:
Virtual Reality
The VR team’s objective was to learn about virtual reality and the basics of VR setting; development, and building in Unity3D. Additionally, the VR Team worked with Oculus Quest 3.0 (or other equivalent devices) as part of creating XR experiences using VR interaction techniques.
Moreover, we develop a web application that is interactive and one that can be used for data exploration from VR platforms. This involved creating a single centralized database that allows easy access to information. For both front-end and back-end components, the team did full-stack development using HTML, CSS, JavaScript and Flask. In the course of the project, they have gained experience in implementing web apps as well as sharpening their ability to work collectively with others on the same team. Moreover, there were also brainstorming sessions which helped in converting client’s use cases into functionalities of the web application.
Artificial Intelligence
We create an XR AI agent, using for instance LLM which is one of large language models in order to realize this. Because it will be able to recognize human emotions expressed on faces and through voices too. To achieve this they familiarize themselves with deep learning/machine learning technology and worked on different types of models such as contemporary CNNs or Transformers among others; Moreover, they explored basics behind machine learning alongside practical instances where these methods were applied in different areas of life.
We also create an emotion recognition model to detect the emotions of users viewing content on a website. The team aimed to test and analyze several machine learning models to select the best-performing one for emotion recognition. The input to the model will be video frames from the user’s camera, and the user’s information will be integrated with their current emotion and the content they are viewing. Optimizing the model for VR Space.
Data Science
We develop a recommendation system using advanced data analytics techniques. First, they used data analytics methods to analyze datasets, picking out important patterns that form the basis for an initial recommendation system. After that, the team concentrated on refining it through intensive user testing. The aim was to fully evaluate and adjust how well the suggestion mechanism worked and how accurate it was from users’ responses or choices. While additionally recommending the most suitable activity to users based on their stress level and type.
Moreover, we develop a visual representation of a conversational interface for communication with virtual reality characters. This involved training the members to analyze and get valuable information from stored chat data by employing techniques such as keyword extraction and summarization.
Research & Design
We examine gender roles in society and explore new gender identities using Queer Theory, Intersectionality, and LGBTQ+ Rights and Identities as lenses. Another objective was finding out how these ideas can be integrated into virtual reality platforms while considering user engagement with such applications. The team will present a research paper that demonstrates how VR environments extend our understanding of this field. Each response given by participants during the study will help in creating more complex simulations. The technical writing report format will be used to blend gender principles with functionalities of VR systems. Additionally, the team designed the framework for a mobile application as part of Project XR, aiming to extend these explorations to a broader audience. This mobile platform will facilitate greater user engagement and accessibility across different devices.
