Gboard Tests Smartphone Camera-Based Sign Language Recognition Feature

Google’s Gboard keyboard is preparing to introduce a new sign language recognition feature, enabling users to input text via gestures captured by their smartphone camera. The technology is built on Google DeepMind’s SignGemma model and employs a hybrid data processing approach: video is analyzed locally on the device, while gesture recognition is performed in Google’s cloud.

The Sign-to-Text feature aims to enhance mobile device accessibility for individuals with hearing and speech impairments. In the current Gboard beta version (17.8.3.939743344), text prompts have been discovered to help users position their hands correctly in front of the camera. For instance, the system will alert users about insufficient lighting and suggest moving to a brighter location.

It remains unclear which sign languages will be supported initially. American Sign Language (ASL) is the most likely candidate, while support for other regional variants, such as British Sign Language, has not been announced. Additionally, compatibility with various smartphone models is uncertain, as video processing demands significant computational resources.

Google emphasizes that raw video footage will not be sent to the cloud—only gesture data. This approach prioritizes user privacy, particularly given the sensitive nature of such data. In the future, this feature could mark a significant step in advancing inclusive technologies, simplifying communication for millions worldwide.