Conversational Interaction Studies
Pilot Study: AAC User / Therapist Interactions

In our preliminary investigation, we analysed videotaped sessions of adolescent word board users describing pictures to their speech therapists. Our goal was to observe patterns of verbal and gestural communication in these sessions, and to consider how they may reflect more general features of augmented conversation. We identified a number of interactional features, including cooperative construction ("co-construction") of sentence meanings, word-finding, and conversational repair, by both participants in each session.


Table of Contents


Patrick Demasco, Kathy McCoy, Denise Peischl, Chris Pennington , Peter Vanderheyden, Wendy Zickus

[Conversational Studies] [NLI] [ASEL]

Lisa Michaud -- michaud@asel.udel.edu
Last modified: Fri Jan 30 15:47:30 EST 1998



Conversation is often a cooperative, bi-directional, and multimodal process of constructing and exchanging information. In the context of AAC, a conversational partner often becomes actively involved in constructing the augmented speaker's message. The partner may ask questions, repeat part of the augmented speaker's utterance, or simply nod and smile in agreement. This feedback may in turn affect the message being produced by the augmented speaker.

Computer-based AAC systems currently "see" only the words the user selects. The less information the user provides as input, the less the likelihood of accurate output. Studies with manual AAC systems suggest, however, that other modes of communication may be preferred. For example, children with cerebral palsy chose to use vocalizations, gestures, or eye gaze as modes of communication far more often than their manual symbol boards in interactions with their mothers or their speech therapists. These alternate modes of communication may be critical to fully understand augmented interactions. Already, the use of gestural recognition is being considered for future AAC systems.


This pilot study is our first step in defining design criteria for an augmentative communication system that provides common interactional features of a conversation involving a person using a communication aid.


Four adolescent students with cerebral palsy and their speech therapists took part in this study. Each student used a manual word board to describe pictures in a storybook to their therapist, as if telling a story to younger children.The therapist was instructed to repeat each word as it was selected, and to paraphrase the student's sentence when it was complete. These sessions were originally recorded for van Balkom et al. (in preparation).

Several modalities of student and therapist communication were transcribed, including vocal productions (both words and non-words), hand and arm gestures, facial expressions, and head gestures.


Interaction and Negotiation

Examining the transcripts, we found a number of interesting patterns of negotiation and interaction between therapist and student.

(a) Repetition by listener. Contrary to instructions, therapists often interpreted students' utterances before the student finished the sentence. The interpretations could be described as varying along two dimensions: incrementality and compansion.

An incremental interpretation is one during which the listener repeats the entire sentence so far. In a compansive interpretation, the listener inflects the student's utterance and adds function words.

In the following example, the student produces the words 'boy', 'girl', and 'walk' (in italics). The therapist (in bold) might respond as follows:

Non-compansive, and non-incremental:
boy boy girl girl walk walk
Non-compansive, and incremental:
boy boy girl boy girl walk boy girl walk
Compansive, and non-incremental:
boy a boy girl and a girl walk are walking
Compansive, and incremental:
boy a boy girl a boy and a girl walk a boy and a girl are walking
(b) Word finding strategies. When a student did not know a word, the student or the therapist often initiated some kind of word finding strategy. Verbal/linguistic strategies observed included selecting a synonym or related word(s), spelling the word, or giving the first letter, or using an idiosyncratic word to signal a lexical gap. As well, gestural strategies, such as iconic gestures or pointing, were employed.

Implications for Compansion

Analysis of the transcripts also revealed information relevant to our AAC design assumptions. Specifically, some of the interpretations that therapists produced for a student's utterances were consistent with our
Compansion system, and some were not.

(a) Transformations consistent with Compansion. Therapists performed "standard" Compansion transformations on students' utterances, such as adding missing function words and appropriate syntactic affixes (e.g. tense markers). As well, Compansion is currently able to perform such observed transformations as changing the word order in a sentence, inferring a missing agent or verb, and adding conjunctions.

(b) Transformations beyond the scope of Compansion. Therapists were able to drop or replace inappropriate words from an utterance, possibly using more than just the linguistic context. They could infer which verb should be used, if no verb was given by the student. In most cases, Compansion cannot perform these transformations.

Therapists were privy to information that was not available to Compansion. For example, in some cases the therapist appeared to be making use of the student's gestures, objects in the room, or pictures in the storybook to disambiguate an utterance, none of which are available to the Compansion system. In other cases, therapists may have used their knowledge of the world to infer what the student had intended to say.


The purpose of this pilot study was to help identify issues relevant to natural conversations and AAC systems for further investigation. As a result, we are preparing a number of subsequent studies, listed in the previous page, to pursue these issues further.


McCoy, K. F., McKnitt, W. M., Peischl, D. M., Pennington, C. A., Vanderheyden, P. B., & Demasco, P. W. (1994) AAC-user therapist interactions: Preliminary linguistic observations and implications for compansion. In M. Binion (Ed.), Proceedings of the RESNA '94 Annual Conference (pp. 129-131). Arlington, VA: RESNA Press.
abstract], [text (13K)], [postscript (75K)]

Vanderheyden, P. B., Pennington, C. A., Peischl, D. M., McKnitt, W. M., McCoy, K. F., Demasco, P. W., van Balkom, H., & Kamphuis, H. (1994). Developing AAC systems that model intelligent partner interactions: Methodological considerations. In M. Binion (Ed.), Proceedings of the RESNA '94 Annual Conference (pp. 126-128). Arlington, VA: RESNA Press.
[abstract], [text (13K)], [postscript (67K)]

van Balkom, H., Kamphuis, H., Demasco, P., & Foulds, R. (in preparation). Language technology in AAC: Automatic translation of graphic symbols into text and/or synthesized speech.


This work has been supported by a Rehabilitation Engineering Center grant from the National Institute on Disability and Rehabilitation Research. Additional support has been provided by the Nemours Foundation.