Conversational design with multi-modalities

Introduction

Nowadays, we are confronting various new machines equipped with intelligence, and they can interact with us socially. Social robots, voice user interfaces, and chatbots are such examples. They show a lot of promise in using conversational interaction to simplify and amplify our everyday personal and professional lives. However, conversational design is more than creating interfaces that can talk or chat. A good conversation is more than an exchange of phrases. It begins with an unspoken agreement and succeeds with cooperation toward a goal. The primary interest of this course is to explore design principles and use them to design intuitive interactions with digital systems and whether a system can use the proper modalities to fit the contexts and activities. Informed by HCI studies on human-human conversations, we already knew that there are essential qualities of a good conversation, including Goal-oriented, Cooperative, Context- aware, Quick and straightforward, Turn-based, Truthful, Polite, and Error-tolerant. In the first part of the course, students will study to understand those conversational qualities and individually develop interaction concepts with a chosen modality, such as voice, text, gesture, haptic, or tangible. The designed interactions should embody the qualities that fit usage scenarios. In the second part, students will use the interaction-attention continuum to analyse the interaction flow in a broader everyday context. Based on the analysis, students will expand the interaction to provide seamless experiences connecting the conversations from one modality to another. This exercise is beneficial because an intelligent system will continuously grow with newly added devices and agents. Designers need a holistic view of how the conversation could fluently switch among agents with alternative modalities. In this way, designers will create seamless conversational interactions to help people reach their goals easily and efficiently.

My role

• 2023–: lecturer.

Learning goals

This course aims to enhance students’ competencies in the following two expertise areas: (1) Creativity and Aesthetics, (2) Technology and Realization. The teaching program will introduce and discuss related theories and methods of designing conversational interactions. Students will apply learned knowledge with the term project and reflect on their practice to improve their design attitude, skill, and understanding. Overall, we break down the goal into the following six objectives. For every lecture and assignment, students can see the mark of the relevant objectives:

• L1. Understanding the essential qualities of a successful conversational design.
• L2. Using the methodological and theoretical knowledge of analysing conversations and creating interaction designs.
• L3. Embodying the conversational qualities in creating proper designs that can fit the interaction context.
• L4. Describing differences between modalities and benefits of multi-modal feedback.
• L5. Creating appropriate user experiences in switching the conversations through different modalities.
• L6. Using AI prototyping tools in crafting good designs with aesthetic qualities.

Course setup

One assignment will be published on Canvas one week before the first meeting. You will be asked to indicate your preference for the projects before the deadline mentioned in the announcement. After receiving students’ registrations, you will be assigned to groups by the course coordinator. The allocation will be published in Canvas before the first meeting. In the first week of the quartile, you can start the project work immediately. The initial design challenges were described in the Project Description section on pp. 6-9.

Course schedule and deliverables

In this course, students will learn several design principles and techniques for creating appropriate experiences of conversational designs. Students will also read literature and apply all the knowledge through practising with a design project. To empower students’ learning-by-doing practice, we integrate the user-centre-design and double-diamond process with the weekly topics and assignments. The overall program is presented in the following diagram, followed by the descriptions of the weekly teaching plan.