Envisioning a new model for voice interaction

Plug+ is a trainable switch that explores an alternative model of voice interaction in future intelligent homes. By connecting with different appliances, it enables people to create their own voice command to control home appliances.


The Context

While products have been designed and manufactured to serve a predefined use, logic, and behavior, with advances in emerging technologies such as machine learning, increasingly objects are acquiring the capability of learning, context-awareness, and evolving affordances. This paradigm shift brings new models of interactions between people and objects. New complexities inside objects also bring challenges for designers to translate and bridge the gap in understanding these interaction.

How can we create new metaphors for interactions with intelligent objects? What language should we use to communicate objects’ learning process? What new values can this kind of trainable intelligence bring to us?





The Concept

Plug+ is an exploration of trainable interfaces for intelligent objects. We use switches, an important yet conventionally passive electrical component of a home, to explore the potential of training in a tangible way. We examined a range of machine learning models such as supervised learning and reinforcement learning to inform the training interface to enable the user to teach it more actively.

The prototype consists of two part, a switch box that allows you to connect any of your appliances, and a detachable training module that you can place anywhere in your home and train it with your unique voice command.




The Interaction

To make the training experience intuitive, we opted for a multimodal interaction – voice interaction in concert with tangible input and visual output. The push button provides a tactile trigger for the training. Once the device is triggered to listen, the LED light changes to remind the user to start recording. The light also provides a direct and understandable feedback to demonstrate the training progress. When the training is completed, the breathing light will appear, indicating the using mode. The user can always retrain Plug+ by repeating the same process.


The Technology

Plug+ is built with Arduino and Wekinator. A push button connected with Arduino allows the user to input a unique voice command. The voice command then is sent to Wekinator via open sound control (OSC) to be classified. To ensure the accuracy, four audio samples are required to complete the training. This system enables the training and using to happen locally, making it more reliable, free from concerns about data privacy in cloud-based systems.  With the modular design, user can connect Plug+ with any appliances and use it in different scenarios.


The Team

Yuxi Liu
Reuben Jerome Dsilva
Varenya Raj

Instructors: Massimo Banzi & Simone Rebaudengo