Seminar on Speech and Language Processing for Augmentative and Alternative Communication

Brian Roark

Richard Sproat

Spring 2010

TR 4-5:20, WCC403

Videoconf'd to BICC 131B, except when noted in the syllabus

Office Hours: TBD


One of Rasmus Malling Hansen's writing balls for the blind (ca. 1865).
Source: image © 2009 by Auction Team Breker, Cologne, Germany (see also The International Rasmus Malling-Hansen Society), used with permission.

Description Syllabus

Description

 This seminar introduces the field of speech and natural language processing (SNLP) as it relates to Augmentative and Alternative Communication (AAC). We have selected a set of representative papers that show how SNLP techniques have been used in AAC. The bulk of the course will consist of participant-led discussions on these papers. There will be one introductory lecture that will introduce AAC and give a brief history of mechanical and electronic technology with AAC applications. Another component of the course will be hands-on experience with AAC devices.

Grading for the course will be based on participation in discussion, and a course project on some topic related to AAC. Possible projects include:

The project will include a presentation (during week 10 and if necessary week 11), and a (maximum) 10-page writeup. Project proposals -- maximum of one page -- are due at the end of the sixth week. The grading will break down as follows:

Project 50%
2 presentations of readings 30%
Class participation 20%

AAC is about helping individuals communicate. An excellent webcast of a lecture by Colin Portnuff called "AAC: A User's Perspective" can be viewed here. We strongly encourage students to view this video within the first week of class, to frame the issues that we will discuss within the real-world applications that are the domain of the topic. You should also read the short introductory chapter to:

D. Beukelman and P. Mirenda. 1998. Augmentative and Alternative Communication: Management of Severe Communication Disorders in Children and Adults. Second Edition. Paul H. Brookes, Baltimore, MD.

The longer Chapter 3 from Beukelman and Mirenda, discusses issues with symbol systems as well some of the prediction algorithms we will be examining in this course.

In addition to readings on speech and language technology, we will also have periodic guest lectures by users of AAC and the clinicians who work with them.

A new webcast by David Beukelman and Susan Fager on "Supporting Communication of Individuals with Minimal Movement" discusses some recent developments in AAC.

Syllabus

Week 1: 3/30, 4/1

Class 1: Lecture; Introduction to course; Scope of AAC technology; Survey of General Areas of AAC technology; History of AAC technology from the 18th century to the present; Issues with Symbol Systems; Assignment of readings.

The remainder of the course will be based on readings. Each class will discuss a set of papers centered around a theme. Discussion will be led by one participant for each class.

Class 2: Text Entry methods. Readings:

  1. Higginbotham, D. Jeffery. 1992. Evaluation of keystroke savings across five assistive communication technologies. Augmentative and Alternative Communication, 8:258–272. Papers/aac_journal04.pdf
  2. G. Lesher, B.J. Moulton and D.J. Higginbotham. Techniques for Augmenting Scanning Communication. Augmentative and Alternative Communication 14, 81-101. 1998. Papers/aac_journal01.pdf
  3. Ward, D. J., Blackwell, A. F., & MacKay, D. J. C. (2002). Dasher: A gesture-driven data entry interface for mobile computing. Human–Computer Interaction, 17, 199–228. Papers/dasher.HCI.pdf

Presenter and discussion lead: Brian Roark (Hroðekr)

Week 2: 4/6, 4/8

 Class 3: Guest lecture, Chris Gibbons, and an AAC user.

Demonstrating the use of the P-switch in a scanning communication setup.

Class 4: Ambiguous keyboards. Readings:

  1. MacKenzie, I. S., & Soukoreff, R. W. (2002). Text entry for mobile computing: Models and methods, theory and practice. Human–Computer Interaction, 17, 147-198. Papers/mackenzie.soukoreff.pdf
  2. Lesher, Gregory, Bryan Moulton, and D. Jeffery Higginbotham. 1998. Optimal character arrangements for ambiguous keyboards. IEEE Transactions on Rehabilitation Engineering, 6:415–423. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.3405&rep=rep1&type=pdf
  3. MacKenzie, I. S. (2009). The one-key challenge: Searching for a fast one-key text entry method. Proceedings of the ACM Conference on Computers and Accessibility – ASSETS 2009, pp. 91-98. Papers/assets2009.pdf

Presenter and discussion lead: Nate Bodenstab

Week 3: 4/13, 4/15

 Class 5: Interface variations. Readings:

  1. T. Wandmacher, J.Y. Antoine, F. Poirier, and J.P. Departe. 2008. Sibylle, an assistive communication system adapting to the context and its user. ACM Transactions on Accessible Computing (TACCESS), 1(1):6:1–30. Papers/wandmacher.pdf
  2. Benjamin Blankertz, Matthias Krauledat, Guido Dornhege, John Williamson, Roderick Murray-Smith, and Klaus-Robert Müller. A note on brain actuated spelling with the Berlin Brain-Computer Interface. In C. Stephanidis, editor, Universal Access in HCI, Part II, HCII 2007, volume 4555 of LNCS, pages 759–768, Berlin Heidelberg, 2007. Springer. Papers/BlaWilMur07.pdf
  3. S. Harada, J.A. Landay, J. Malkin, X. Li and J.A. Bilmes. The Vocal Joystick: Evaluation of voice-based cursor control techniques for assistive technology. Disability and Rehabilitation: Assistive Technology, January/March 2008; 3(1 – 2): 22 – 34. Papers/vocaljoystickeval.pdf

Presenter and discussion lead: Eric Morley

Class 6: Language modeling and word prediction. Readings:

  1. Tonio Wandmacher; Jean-Yves Antoine. Methods to Integrate a Language Model with Semantic Information for a Word Prediction Component. EMNLP/CoNLL, 2007, pp. 506-513. http://www.aclweb.org/anthology-new/D/D07/D07-1053.pdf
  2. Trnka, Keith, John McCaw, Debra Yarrington, Kathleen F. McCoy, and Christopher Pennington. User Interaction with Word Prediction: The Effects of Prediction Quality. ACM Transactions on Accessible Computing (TACCESS), 1(3), 2009, pp. 1-34. Papers/a17-trnka.pdf
  3. Tam, C., Reid, D., Naumann, S., O'Keefe, B., 2002. Effects of word prediction and location of word prediction list on text entry with children with spina bifida and hydrocephalus. Augmentative and Alternative Communication. 18 (September), 147--162. Papers/7574632.pdf
  4. Higginbotham, D. J., Bisantz, A. M., Sunm, M., Adams, K., Yik, F., 2009. The effect of context priming and task type on augmentative communication performance. Augmentative and Alternative Communication 25 (1), 19--31. Papers/x2009_05.pdf

Presenter and discussion lead: Mahsa Yar Mohammadi

Week 4: 4/20, 4/22

 Class 7: Coding and information theoretic perspectives. Readings:

  1. M. Baljko and A. Tam. Indirect Text Entry Using One or Two Keys. 2006. In Proceedings of the Eighth International ACM Conference on Assistive Technologies, ASSETS'06, pp. 18--25. Papers/p18-baljko.pdf
  2. Brian Roark. 2009. Open vocabulary language modeling for binary response typing interfaces. Technical Report #CSLU-09-001, Center for Spoken Language Processing, Oregon Health & Science University. http://www.cslu.ogi.edu/publications/ps/roark09.pdf
  3. M. Baljko. The Information-Theoretic Analysis of Unimodal Interfaces and their Multimodal Counterparts. 2005. In Proceedings of the Seventh International ACM Conference on Assistive Technologies, ASSETS'05, pp. 28--35. Papers/baljko05b.pdf
  4. Terence D. Sanger and Juliet Henderson. Optimizing Assisted Communication Devices for Children With Motor Impairments Using a Model of Information Rate and Channel Capacity. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15(3), 2007, pages 458-468. Papers/channelcapacity.pdf

Presenter and discussion lead: Emily Tucker Prud'hommeaux

Class 8: Utterance-based systems. Readings:

  1. McCoy, Kathleen F., Jan L. Bedrosian, Linda A. Hoag, and Dallas E. Johnson. 2007. Brevity and speed of message delivery trade-offs in augmentative and alternative communication. Augmentative and Alternative Communication, 23(1):76–88. Papers/24154781.pdf
  2. Todman, John, Norman Alm, D. Jeffery Higginbotham, and Portia File. 2008. Whole utterance approaches in AAC. Augmentative and Alternative Communication, 24(3):235–254. Papers/aac_journal02.pdf
  3. Wisenburn, B., Higginbotham, D. J., 2008. An AAC application using speaking partner speech recognition to automatically produce contextually relevant utterances: Objective results. Augmentative and Alternative Communication 24 (2), 100-109. Papers/31937749.pdf

Presenter and discussion lead: Maider Lehr

Week 5: 4/27, 4/29

 Class 9: Symbol Systems. Readings:

  1. Schlosser, R. W. and Sigafoos, J., 2002. Selecting graphic symbols for an initial request lexicon: Integrative review. Augmentative and Alternative Communication. 18(June), 102-123. Papers/6860021.pdf
  2. P. L. Albacete, S. K. Chang and G. Polese. 2006. Iconic language design for people with significant speech and multiple impairments. Assistive Technology and Artificial Intelligence. Lecture Notes in Computer Science, 1458. Springer. Papers/albacete.pdf
  3. Binger, C., Light, J., 2007. The effect of aided AAC modeling on the expression of multi-symbol messages by preschoolers who use AAC. Augmentative and Alternative Communication 23 (1), 30-43. Papers/24154784.pdf

Class canceled

Class 10: More Symbol Systems. Readings:

  1. Mizuko, M. 1987. Transparency and ease of learning of symbols represented by Blissymbolics, PCS, and Picsyms. Augmentative and Alternative Communication, 3:129–136. Papers/aac_journal03.pdf
  2. Cohen, E. T., Allgood, M. H., Heller, K. W., Castelle, M., 2001. Use of picture dictionaries to promote written communication by students with hearing and cognitive impairments. Augmentative and Alternative Communication 17 (December), 245-254. Papers/5566283.pdf
  3. Ganz, J. B., Sigafoos, J., Simpson, R. L., Cook, K. E., 2008. Generalization of a pictorial alternative communication system across instructors and distance. Augmentative and Alternative Communication 24 (2), 89-99. Papers/31937750.pdf

Presenter and discussion lead: Masoud Rouhizadeh

Week 6: 5/4, 5/6

Project proposals (maximum 1 page) due on Friday 5/7

Class 11: Yet More Symbol Systems. Readings:

  1. Patel, R., Pilato, S., & Roy, D. (2004). Beyond Linear Syntax: An Image-Orientation Communication Aid. Assistive Technology Outcomes and Benefits, 1(1), 57-67. Papers/patel.ATOBV1N1.pdf
  2. Hamed H. Sad and Franck Poirier. Using Pictographic Representation, Syntactic Information and Gestures in Text Entry. Human-Computer Interaction, Part II, HCII 2009, LNCS 5611, pp. 735–744, 2009. Papers/sadpoirer09.pdf
  3. Sutton, A., Soto, G., Blockberger, S., 2002. Grammatical issues in graphical symbol communication. Augmentative and Alternative Communication 18 (September), 192-204. Papers/7575188.pdf

Class 12: Yet More Symbol Systems; Relevant User Issues Papers. Readings:

  1. Patel, R., Schooley, K. & Radhakrishnan, R. (2006). Comparison of semantic versus syntactic message formulation: A pilot study. Assistive Technology Outcomes and Benefits, Volume 3, 96-110. Papers/patel.ATOBV3N1.pdf
  2. Ottem, E., 2001. Use of pictographic-articulatory symbols to promote alphabetic reading in a language-impaired boy: Case study. Augmentative and Alternative Communication 17 (March), 52-60. Papers/4546001.pdf
  3. Venkatagiri, H. S., 2002. Clinical implications of an augmentative and alternative communication taxonomy. Augmentative and Alternative Communication 18 (March), 45-57. Papers/6686902.pdf

Presenter and discussion lead: Mahsa Yar Mohammadi

Week 7: 5/11, 5/13

 Class 13: Guest Lecture: Janice Staehely and Melanie Fried-Oken, 3-5pm at CDRC (room 3200)

Class 14: Speech Transformation, Recognition and Assessment Papers. Readings:

  1. A. Kain, J. Hosom, X. Niu, J. van Santen, M. Fried-Oken, J. Staehely. Improving the Intelligibility of Dysarthric Speech. Speech Communication, Volume 49, Issue 9, September 2007, Pages 743-759. Papers/kain.SC.pdf
  2. A. Kain, J. van Santen. Using Speech Transformation to Increase Speech Intelligibility for the Hearing- and Speaking-impaired. Proceedings of ICASSP, April 2009. http://www.cslu.ogi.edu/~kain/pub/Kain2009-ICASSP-Transformation.pdf
  3. H. Timothy Bunnell, Chris Pennington, Debra Yarrington, and John Gray. Automatic personal synthetic voice construction. Interspeech 2005. Papers/bunnell05.pdf

Presenter and discussion lead: Maider Lehr

Week 8: 5/18, 5/20

 Class 15: Guest Lecture: Alex Kain, Speech Transformation

Class 16: Speech Recognition and Assessment Papers. Readings:

  1. M. Hasegawa-Johnson, J. Gunderson, A. Perlman, T. Huang. HMM-Based and SVM-Based Recognition of the Speech of Talkers with Spastic Dysarthria. ICASSP, May 2006. http://www.isle.illinois.edu/pubs/2006/hasegawa-johnson06icassp.pdf
  2. A. Tsanas, M. A. Little, P. E. McSharry, L. O. Ramig. Enhanced Classical Dysphonia Measures and Sparse Regression for Telemonitoring of Parkinson's Disease Progression. ICASSP 2010. http://www.eng.ox.ac.uk/samp/members/max/publications/tsanas_icassp2010.pdf
  3. G. E. Lancioni, M. F. O'Reilly, N. N. Singh, J. Sigafoos, D. Oliva, G. Montironi, M. Savino and A. Bosco. Extending the evaluation of a computer system used as a microswitch for word utterances of persons with multiple disabilities. Journal of Intellectual Disability Research, 49(9):639-646, 2005. Papers/lancioni05.pdf

Presenter and discussion lead: Tomer Meshorer

Week 9: 5/25, 5/27

 Class 17: Natural Language Generation Papers. Readings:

  1. E. Reiter, R. Turner, N. Alm, R. Black, M. Dempster and A. Waller. Using NLG to help language-impaired users tell stories and participate in social dialogues. In 12th European Workshop on Natural Language Generation (2009), Association for Computer Linguistics, pp.1-8. Papers/p1-reiter.pdf
  2. Yael Netzer and Michael Elhadad. 2006. Using Semantic Authoring for Blissymbols Communication Boards. In Proceedings of Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. http://www.cs.bgu.ac.il/~yaeln/papers/naacl06.pdf
  3. A. Waller, R. Black, D. O'Mara, H. Pain, G. Ritchie, R. Manurung. Evaluating the STANDUP pun generating software with children with cerebral palsy. ACM Transactions on Accessible Computing (TACCESS), 1(3) (2009), pp.16:0-16:27. Papers/a16-waller.pdf

Presenter and discussion lead: Tomer Meshorer

Class 18: User Issues Papers. Readings:

  1. Beukelman, D. R., Fager, S., Ball, L., Dietz, A., 2007. AAC for adults with acquired neurological conditions: A review. Augmentative and Alternative Communication 23 (3), 230-242. Papers/26287727.pdf
  2. Binger, C., Light, J., 2008. The morphology and syntax of individuals who use AAC: Research review and implications for effective practice. Augmentative and Alternative Communication 24 (2), 123-138. Papers/31937747.pdf
  3. Bourgeois, M. S., Dijkstra, K., Burgio, L., Allen-Burge, R., 2001. Memory aids as an augmentative and alternative communication strategy for nursing home residents with dementia. Augmentative and Alternative Communication 17 (September), 196-210. Papers/5469209.pdf

Presenter and discussion lead: Masoud Rouhizadeh

Week 10: 6/1, 6/3

 Class 19: Testing/Evaluation Papers. Readings:

  1. File, P., Todman, J., 2002. Evaluation of the coherence of computer-aided conversations. Augmentative and Alternative Communication 18 (December), 228-241. Papers/8833016.pdf
  2. Todman, John and Halina Rzepecka. 2003. Effect of pre-utterance pause length on perceptions of communicative competence in AAC-aided social conversations. Augmentative and Alternative Communication, 19(4):222–234. Papers/aac_journal06.pdf
  3. Ball, L. L., Beukelman, D. R., Pattee, G. L., 2004. Acceptance of augmentative and alternative communication technology by persons with amyotrophic lateral sclerosis. Augmentative and Alternative Communication 20 (2), 113-122. Papers/13309375.pdf

Presenter and discussion lead: Eric Morley

Class 20: User Issues Papers. Readings:

  1. Light, J., Drager, K., 2007. AAC technologies for young children with complex communication needs: State of the science and future research directions. Augmentative and Alternative Communication 23 (3), 204-216. Papers/26287729.pdf
  2. E.W. Sellers and E. Donchin. 2006. A P300-based brain-computer interface: initial tests by ALS patients. Clinical Neurophysiology, 117:538-548. Papers/sellers.pdf
  3. Sigafoos, J., Green, V. A., Payne, D., Son, S.-H., O'Reilly, M., Lancioni, G. F., 2009. A comparison of picture exchange and speech-generating devices: Acquisition, preference, and effects on social interaction. Augmentative and Alternative Communication 25 (2), 99--109. Papers/x2009_09.pdf

We will not discuss these papers in class, but you should read them anyway.

The class activity will be Maider Lehr's presentation (30 minutes)

Week 11 (finals week): 6/8

© 2010, Richard Sproat and Brian Roark