Speech and Sketching

Speech and Sketching: An Empirical Study of Multimodal Interaction - A. Adler and R. Davis

Adler and Davis conduct a series of user studies to determine how a user would interact with a system combining both sketch and speech input that moves towards an interactive, 2-way communication between system and user. Previous studies were often limited to a set of commands, one-way interaction, annotation (no sketching), or a set of fixed sketch symbols. Rather than attempting to set up a complex Wizard of Oz experiment, users and experimenters sat face to face with the experimenter taking the role of an idealized recognition system and responded to the user's verbal questions and modifying drawings as appropriate. Users were asked to sketch and describe drawings from four different domains. The recorded conversations were analyzed qualitatively in five areas: sketching, language, multimodal interaction, questions, and comments. Sketching was divided into groups by stroke and amount of ink. By stroke, 52% were creative, 40% writing, 5% selection, and 3% modification, while by amount of ink, 63% was from creation, 21% writing, 8% selection, and 8% modification. Three colors were used on average and users frequently changed colors. Color was used to refer to previously sketched sections, to create groups, or to reflect the real-life color of a sketched object. They found language was often ungrammatical and repetitive, with the repeated phrases being key to the discussion, and related to what the user was currently sketching. Multimodal interaction often linked the order of speaking and drawing or combined speaking an annotation or label with saying it. Users often completed a sketch that was spoken about before speaking again. When asked questions, the users clarified the sketch and were encouraged to speak more freely. When commenting the user often described what they were planning to do or made comments about the sketch quality. Qualitatively, users were found to begin speaking before drawing if they were speaking a phrase, but to begin drawing before speaking if they were speaking a single word.