AI-Powered In-Game Sign Language Translator Unveiled by Sony

Sony has filed a patent that aims to provide increased accessibility for deaf gamers. Sony's patent demonstrates how specific sign languages can be translated to another in-game.

Sony Patents ASL to JSL Translator for Video GamesProposed to Use VR Devices and Work Over Cloud Gaming

Sony has filed a patent that adds a real-time sign language translator to video games. The patent, titled "TRANSLATION OF SIGN LANGUAGE IN A VIRTUAL ENVIRONMENT," illustrates a cutting-edge technology wherein American Sign Language (ASL) could be communicated to a Japanese-speaking user using Japanese Sign Language (JSL).

Sony said it aims to put in place a system that can assist deaf gamers via real-time translations of sign languages during in-game conversations. The technology described in the patent would allow virtual indicators or avatars that are displayed on-screen to communicate sign language in real-time. The system will initially translate the sign gestures of one language into text, then convert the text to another specified language, and finally translate the data received into the sign gestures of the other language.

"Implementations of the present disclosure relate to methods and systems for capturing sign language of one user (e.g., Japanese), and translating the sign language to another user (e.g., English)," Sony described in the patent. "Because sign languages vary depending on geographical origins, sign language is not universal. This provides a need for appropriately capturing the sign language of one user, understanding the native language, and generating new sign language as output for another user in their native sign language."

One way this system could be implemented, as Sony illustrated, would be with the aid of a virtual reality-type device or head-mounted display (HMD). "In some implementations, the HMD connects via a wired or wireless connection to a user device, such as a personal computer, game console, or other computing device," Sony elaborated. "In some implementations, the user device renders graphics for display through the HMD that provide engrossing viewing of the virtual environment for the user."

Sony further proposed that one user device could seamlessly communicate with another user device over a network with a game server. "In some implementations, the game server executes a shared session of a video game, maintaining the authoritative state of the video game and its virtual environment," Sony said, "and to which the user devices are synchronized regarding the state of the virtual environment."

With this setup, users can share and interact with each other in the same virtual environment, aka game, over a shared network or server. Sony additionally said that in some implementations of the system, the game server could be part of a cloud gaming system, which "renders and streams video" between each user device.