hct logo part1UBC
department of ECE
HCT title
about_hctresearchpeopleopportunitiespublicationsresourcescontact
side menu top
music and sound
art and performance
physical interfaces
modeling
graphics
side menu end
hct logo part2
Glove-TalkII
An Adaptive Interface that Maps Hand Gestures to Speech
GloveTalkII Graphic A Project by
Sidney Fels


Abstract
Downloads
- Videos
Publications
Contact Information

GloveTalkII is a system that translates hand gestures to speech through an adaptive interface. Hand gestures are mapped continuously to ten control parpameters of a parallel formant speech synthesizer. The mapping allows the hand to act as an artificial vocal tract that produces speech in real time. This gives an unlimited vocabulary in addition to direct control of fundamental frequency and volume. Currently, the best version of Glove-TalkII uses several input devices (including a Cyberglove, a ContactGlove, a three space tracker, and a foot pedal), a parallel formant speech synthesizer, and three neural networks. One subject has trained to speak intelligibly with Glove-TalkII. He speaks slowly but with far more natural sounding pitch variations than a text-to-speech syntesizer.

Downloads

Videos
Initial GTII Vocabulary

The initial Glove-TalkII gesture mapping. (28M)

Alphabet

Sageev Oore saying the alphabet. (20M)

Numbers

Sageev Oore saying the numbers up to 20. (18M)

Green Eggs and Ham

Sageev Oore reciting part of Green Eggs and Ham (Dr. Seuss). (46M)

Row, Row, Row your boat

Sageev Oore singing Row, Row, Row your boat.(10M)

Unrehearsed conversation

Sageev Oore and Sidney Fels in conversation (unrehearsed). (183M)

Publications

bibtexSidney S. Fels. Radial Basis Function Networks 2, New Advances in Design. Using Radial Basis Functions to Map Hand Gestures to Speech. Pages 59-101. Howlett, R. J. and Jain, L. C.Physica-Verlag. 2001.
PDFbibtexSidney S. Fels and Geoffrey E. Hinton. Glove-TalkII: A neural network interface which maps gestures to parallel formant speech synthesizer controls. IEEE Transactions on Neural Networks. Volume 9. No. 1. Pages 205-212. 1998.
PDFbibtexSidney S. Fels. Neural Networks for Computer-Human Interfaces: Glove-TalkII. Proceedings of the International Conference on Neural Information Processing (ICONIP96). Pages 1299-1304. Hong Kong. Sept. 1996.
bibtexSidney S. Fels and Goeffrey E. Hinton. Glove-TalkII: An Adaptive Gesture-to-Formant Interface. Proceedings of Computer Human Interaction 1995 (SIGCHI95). Pages 456-463. Denver, CO. May. 1995.
bibtexSidney S. Fels and Goeffrey E. Hinton. Glove-Talk II: Mapping Hand Gestures to Speech Using Neural Networks. Advances in Neural Information Processing Systems (NIPS7). Volume 7. Pages 843-850. Morgan Kaufmann. San Mateo. 1995.
PDFbibtexSidney S. Fels and Goeffrey E. Hinton. Glove-Talk: A neural network interface between a Data-Glove and a Speech Synthesizer. IEEE Transactions on Neural Networks. Volume 4. Pages 2-8. 1993.
bibtexSidney S. Fels. Glove-TalkII: Mapping Hand Gestures to Speech Using Neural Networks - An Approach to Building Adaptive Interfaces. Ph.D. Thesis. University of Toronto. Aug. 1994.

Contact Information

Sid Fels


Last up-dated: 07/23/2002
© 2002-2005 HCT
about_hctresearchpeopleopportunitiespublicationsresourcescontact