Sophia Linguistica, Vol. 50, pp. 221-233, 2003 (in Japanese).
A study on educational tools with an animated talking agent for the hearing impaired
T. Kitamura, T. Arai, P. Connors, P. Stone, K. Iitaka and M. Shindo
Abstract: An education tool with an animated talking agent named Baldi, that has been developed at Center for Spoken Language Understanding at the Oregon Graduate Institute is used at a school for deaf children using oral language in Tucker-Maxon (Portland, Oregon, USA). Tutors for the language education like the CSLU Toolkit have not been used in Japan. In order to implement a Japanese language animated agent, we have tried to map visemes of Mexican Spanish phonemes to the corresponding visemes of Japanese phonemes. In this thesis, we proposed the ways in which these resources can be used effectively in Japan. We focused on the following areas:
- We can obtain not only audible information but also visible information from an animated talking agent;
- By combining an animated talking agent, speech synthesis, speech recognition, and images, we can make any kind of tutor with the CSLU Toolkit.
The result of the perceptual experiment showed that the intelligibility of words was improved by displaying the animation simultaneously. We concluded that the CSLU Toolkit could be effective for training the hearing impaired in Japan.
[PDF (933 kB)]