The Potential of Virtual Reality for Deaf Accessibility

What is Virtual Reality?

Virtual Reality, or VR, is a three-dimensional, computer-generated environment. Participants access VR through a helmet or specially designed glasses. They can move in the space and see other people and locations as if they were real. Unlike traditional video games, in VR, you become a participant in the virtual world.

Current & Potential Uses

While VR is in its infancy, there are some emerging uses for the deaf and hard of hearing on the horizon. Within education and entertainment, we are witnessing an increasing utilization of VR usage.

Sign Language Generation

Many companies in the technology industry are developing the ability to convert text input to sign languages from many different countries, such as ASL, Auslan, NZSL, BSL, and many others. Most will start with a Deaf signer in a mocap suit. The sensors on the suit send data input to the computer where it is then cleaned up by animators. Signs can then be added to a dataset and pulled for use automatically with the assistance of AI and programming.

Sign Language Recognition

Sign language recognition, or SLR, is a rapidly growing technology that offers the potential for deaf and hard of hearing gamers to communicate in VR. Researchers at companies such as Go Sign AI are building a lengthy data set of signs from many deaf users to train large language models on signs. Sign recognition can convert that data to text or audio for other people.

Sign Language Classes

One new use of VR is for learning sign language. The three-dimensional perspective of VR makes it an ideal environment for learning, as it allows learners to see all the movements and handshapes. We used to learn sign language from books that offered a one-dimensional perspective. Since the creation of technology and computers, many are now learning sign language from videos, which has improved the dimension to 2D. However, VR ASL lessons would be a step forward for new learners, providing a more immersive and interactive learning experience.

The ideal VR experience would be interactive. You can watch the avatar sign and copy it. The program can recognize your movements and give corrective feedback in real-time. One notable VR lab pioneering this work is Motion Light Lab, an award-winning research and development lab at Gallaudet University in Washington, D.C. They have completed various projects, from signing avatars for children’s books and TV shows to 3D ASL interactive lessons, and they lead the research on the latest innovations.

They have a program called Signing Avatars Immersive Learning (SAIL) for ASL lessons, which received financial support from the National Science Foundation. Through deep learning programming, users can quickly learn if they complete a sign correctly and gain feedback through their app, “ASL Champ.”

Accessible TV Shows and Games

Within Meta, finger tracking during gameplay was developed in 2022, allowing players to create realistic signs in the VR environment. As mentioned earlier, signing players will likely interact with other players in their native language in the game space soon.

Other potential uses are a real-time interpretation of Deaf communication, transitioning between ASL and English, and vice versa. Imagine a game where you can select the language of your characters to match your native language. Video games are rarely captioned, but this could pave the way for more content and entertainment accessible to the deaf and hard of hearing.

VR avatars can also be used for TV programming, especially cartoons. One example is a children’s TV show, Emma Memma, with a VR avatar that uses Auslan Sign Language. This increases accessibility as her character switches from herself to the cartoon version.

Another example is Avatar: The Way of the Water, which relied heavily on Deaf actor CJ Jones to create a made-up sign language for that world. The animated characters would use their signs to communicate while underwater, which is called Na’vi Sign Language.

Conclusion

The past and future development of virtual reality brings many exciting possibilities for accessibility for the deaf and hard of hearing communities. From TV shows, classrooms, video games, and communication, the opportunities ahead are numerous. Within ten to fifteen years, we could have a new virtual world where everyone is on an equal playing field for the first time.