Conversational Interaction Studies

In our preliminary investigation, we analysed videotaped sessions of adolescent word board users describing pictures to their speech therapists. Our goal was to observe patterns of verbal and gestural communication in these sessions, and to consider how they may reflect more general features of augmented conversation. We identified a number of interactional features, including cooperative construction ("co-construction") of sentence meanings, word-finding, and conversational repair, by both participants in each session.

Table of Contents Background Purpose Method Progress Results Publications Acknowledgements

Contributors Patrick Demasco, Kathy McCoy, Denise Peischl, Chris Pennington , Peter Vanderheyden, Wendy Zickus

Conversation is often a cooperative, bi-directional, and multimodal process of constructing and exchanging information. In the context of AAC, a conversational partner often becomes actively involved in constructing the augmented speaker's message. The partner may ask questions, repeat part of the augmented speaker's utterance, or simply nod and smile in agreement. This feedback may in turn affect the message being produced by the augmented speaker.

Computer-based AAC systems currently "see" only the words the user selects. The less information the user provides as input, the less the likelihood of accurate output. Studies with manual AAC systems suggest, however, that other modes of communication may be preferred. For example, children with cerebral palsy chose to use vocalizations, gestures, or eye gaze as modes of communication far more often than their manual symbol boards in interactions with their mothers or their speech therapists. These alternate modes of communication may be critical to fully understand augmented interactions. Already, the use of gestural recognition is being considered for future AAC systems.

This pilot study is our first step in defining design criteria for an augmentative communication system that provides common interactional features of a conversation involving a person using a communication aid.

Four adolescent students with cerebral palsy and their speech therapists took part in this study. Each student used a manual word board to describe pictures in a storybook to their therapist, as if telling a story to younger children.The therapist was instructed to repeat each word as it was selected, and to paraphrase the student's sentence when it was complete. These sessions were originally recorded for van Balkom et al. (in preparation).

Several modalities of student and therapist communication were transcribed, including vocal productions (both words and non-words), hand and arm gestures, facial expressions, and head gestures.

Interaction and Negotiation

(a) Transformations consistent with Compansion. Therapists performed "standard" Compansion transformations on students' utterances, such as adding missing function words and appropriate syntactic affixes (e.g. tense markers). As well, Compansion is currently able to perform such observed transformations as changing the word order in a sentence, inferring a missing agent or verb, and adding conjunctions.

(b) Transformations beyond the scope of Compansion. Therapists were able to drop or replace inappropriate words from an utterance, possibly using more than just the linguistic context. They could infer which verb should be used, if no verb was given by the student. In most cases, Compansion cannot perform these transformations.

Therapists were privy to information that was not available to Compansion. For example, in some cases the therapist appeared to be making use of the student's gestures, objects in the room, or pictures in the storybook to disambiguate an utterance, none of which are available to the Compansion system. In other cases, therapists may have used their knowledge of the world to infer what the student had intended to say.

The purpose of this pilot study was to help identify issues relevant to natural conversations and AAC systems for further investigation. As a result, we are preparing a number of subsequent studies, listed in the previous page, to pursue these issues further.

McCoy, K. F., McKnitt, W. M., Peischl, D. M., Pennington, C. A., Vanderheyden, P. B., & Demasco, P. W. (1994) AAC-user therapist interactions: Preliminary linguistic observations and implications for compansion. In M. Binion (Ed.), Proceedings of the RESNA '94 Annual Conference (pp. 129-131). Arlington, VA: RESNA Press.
[abstract], [text (13K)], [postscript (75K)]

Vanderheyden, P. B., Pennington, C. A., Peischl, D. M., McKnitt, W. M., McCoy, K. F., Demasco, P. W., van Balkom, H., & Kamphuis, H. (1994). Developing AAC systems that model intelligent partner interactions: Methodological considerations. In M. Binion (Ed.), Proceedings of the RESNA '94 Annual Conference (pp. 126-128). Arlington, VA: RESNA Press.
[abstract], [text (13K)], [postscript (67K)]

van Balkom, H., Kamphuis, H., Demasco, P., & Foulds, R. (in preparation). Language technology in AAC: Automatic translation of graphic symbols into text and/or synthesized speech.

This work has been supported by a Rehabilitation Engineering Center grant from the National Institute on Disability and Rehabilitation Research. Additional support has been provided by the Nemours Foundation.