Project Description

This work proposes a multimodal framework for the intercommunication between blind and hearing impaired people. The algorithms that are developed are based onthe concept of situated modality replacement, which is the use of information originating from various modalities to compensate for the missing input modality of the system or the users. The proposed framework features sign language recognition andsynthesis, speech recognition and synthesis and haptic interaction in virtual environments, all integrated in a virtual treasure hunting game, where the blind and the hearing-impaired users have to collaborate so as to navigate in the virtual environmentand solve the riddle of the game. The treasure hunting game serves both as a multimodal entertainment tool and as an educational game training and familiarizing the users with new assistive technologies. Usability evaluation of the system has shown that the framework has significant potential for disabled users.

K. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, “Using Modality Replacement to Facilitate Communication Between Blind and Hearing Impaired People”, IEEE Multimedia, vol. 13, no. 3, pp. 92-100, July 2006.