Google's Teachable Machine
Open Broadcase Software

UX Research
Machine Learning Training
UX Design

Yuran Ding
Jenna Kim
Taery Kim
Nandini Nair
Stefania La Vattiata

Fall 2020
4 weeks

Caring for one another during pandemic through responding to dynamic contexts on Zoom. 

UnFolding is a project that enables an emotions-aware, caring environment on Zoom through a paper-folding exercise involving journaling. It provides the tools for facilitating or participating in a virtual self-reflection workshop, one that encourages participants to unfold their emotions as they fold paper into various shapes.

My Contribution

I led the UX design and co-led UX research. This includes strategizing our concept development process and developing experience blueprints. I also assisted with the technical part of this project. Specifically, I trained Google’s Teachable Machine to recognize participant’s paper folding stage and helped connected p5.js with OBS camera and Zoom.


Brief Ask
Create a system for Zoom workshops that responds to dynamic contexts and interactions.

Criteria for Success
Our product / digital experience should bring social and emotional values to existing transactional functions of online video communications.
How Might We ...

Relief the loneliness of young students during the pandemic and increase their literacy in visceral feelings through online video-chatting (in particular, Zoom) workshops?

Research & Process


Secondary Research

Impact/Effort Analysis
Literature Review
Think Aloud Analysis
Experience Blueprint

Training Teachable Machine for Interaction
Visual System

00. Inspiration

While loneliness among young people during the pandemic has become a major concern, present Zoom conditions for social communication and exchange of emotions are limited.

News articles focusing on loneliness from COVID-19

We explored tangible forms of counseling therapy, including Eliza, the Rogerian Therapist, which surprised its creator by the number of users who attributed human-like feelings to the computer program that parroted back what users said in the form of questions, and Later Date, which formed connections bsed on question and answers between the artist Lauren McCarthy and strangers. Inspired by these work, we decided to increase literacy in visceral feelings and dealing with loneliness through designing a workshop, because workshop would provide an accessible and friendly way for exchange of emotions in real-time.

01. How Might We

How might we relief the loneliness of young students during the pandemic and increase their literacy in visceral feelings through online video-chatting (in particular, Zoom) workshops?

02. Design Strategy

Considering the main objective of the project, we decided that our main keywords are communication, collaboration, and interaction. In addition to the lack of social and emotional literacy in everyday Zoom classrooms, another downside of Zoom interaction is that everything is flat (screen-based).

Is there a way to provide for an interaction that is more than face to face? To incorporate object? To make interaction more dynamic and expressive? To augment reactions and responses from people during discussions?

As a form of collaborative exercise on mindfulness that yields unique communication and interaction among participants, we chose the activity of origami (paper-folding) as a medium for the workshop and explored ways to capitalize on its benefits when doing it with a group, including creating a sense of community (social connection), yielding a relaxing environment, and boosting the group's morale.

Origami stands out on the impact/effort matrix.

03. Literature Review

We utilized the method of literature review as part of our user research on people who attend therapeutic counseling sessions and engage in other practices of mindfulness. We found that the benefits of self-awareness include proactivity, boost in self-acceptance, encouragement for positive self-development (Sutton, 2016); multiple perspectives, ability to practice self-control, work creatively and productively, and experience pride in self (Silvia & O'Brien, 2004); better decision-making (Ridley, Schutz, Glanz and Weinstein, 1992); and better communications in the workspace, enchancement of self-confidence and job-related wellbeing (Sutton, Williams and Allinson, 2015). We focused on the 7 pillars of insight that help self-awareness, which are: values, passions, aspirations, fit, patterns, reactions, and impact (Eurich, 2018).

Based on these pillars and following key principles of counseling psychology—start from more closed-ended questions and transition to questions that are open, neutral, concise, and balanced—, we formed a curated list of open narratives from negative to positive.

04. Development

Curating the Questions (Iteration 1)

We created our first iteration: a flow of reflection questions and paper folding guidance accompanied by visual audio outputs.

The questions are brought up by facilitators and the participant would perform the paper folding activity while reflecting.

The flow of the questions occur in three stages: describing the day, analyzing and deconstructing the reasons for stress, and recognizing the positives. Each question mapped to a set of insight that help self-awareness. We then mapped the questions for reflection to a set of paper-folding instructions.

Iteration 1:  the journey that maps the reflection questions to a set of insight, and maps the reflection questions to paper-folding instructions.

In the specific context of a university classroom, our users of unfolding are instructors and college students who engage in the integrated activity of paper-folding and journaling. The instructors would be given a tool for leading a therapeutic group workshop and tracking the progress of the participants. The students would be provided with an opportunity to self-reflect, connect with the instructor and other classmates on a deeper, emotional level.

While the synchronous partaking in the paper-folding instills a feeling of togetherness, the self-reflection part of the activity is based on introspections occurring within the individual spaces of the participants. To amplify aspects of collectivism annd collaboration, we decided to incorporate audio to create a chorus-like output at the end of the activity.
Think Aloud Analysis

We conducted think aloud analysis on 6 participants using our iteration 1.  

Pain point 1:
Participants were often shy and embarrassed to share their visceral and personal feelings to their fellows.

Solution 1:
To create a more comfortable and encouraging atmosphere for sharing, we designed our system in a way that participants can represent their feelings using colors — since colors are less specific but can convey the overall feelings well.

Pain point 2:
The interaction between participants and facilitator is somewhat undetermined and a bit too transactional.

Solution 2:
We created a experience blueprint to understand the essence of the whole experience of the workshop and to gauge the interactions between the participants and between the participants and facilitator.

Experience Blueprint
The interactions between the facilitator and participants would be twofold:
1) there is a start, mid-way, and final checkpoint when the participant are asked to hold up a color that best matches their current mood;
2) there are two sets of paper-folding steps (that the facilitator demonstrates), which each triggers a self-reflection question that the participants answer by journaling on their origami-in-progress.
The exercise ends when all participants have finished journaling and folding their origamis, and there is one final collective visual and audio response.

We carefully considered the roles of each part of the interaction system: is the facilitator simply a human guide to the activity or is she an integral part of the interaction system? What information do the participants provide and what are their takeaways? How can we help the facilitator better understand each participant's progress? How can we better communicate each participant's emotions to each other?

Facilitator's role:
  1. guides the activity of paper-folding (each step triggers a question for self-reflection); 
  2. tracks the participants' progress and guages when to move forward or provide more time; 
  3. tracks the participants' emotional states throughout the activity in 3 checkpoints.

Participants' role: 
  1. follows along the facilitator in the activity of paper-folding; 
  2. journals on their folded paper (at each step) prompted by the self-reflection questions; 
  3. provides information about their current emotional state.
Challenge: Tracking Participant Progress Using Teachable Machine
Currently, there are still some challenges for the facilitator to track the participant and also for the participants to provide information about their current state because the signals for these are not so clear on Zoom, particularly in a gallery grid.

Because all of these signals requires image recognition and classification, we decided to use Google's Teachable Machine to read facilitator's and participants' input and provides corresponding visual and audio feedback.

05. Interaction

We focused on designing for interaction that augmennts emotions-based expressions and reactions. We trained two image models using Teachable Machine, one using 9 different classes for the facilitator and another using 6 different classes for the participants. They each register a sequence of paper-folding steps and a scale of emotions (using color). From the assigned roles (above), we determined that a set of tools is needed both for the interactions among the participants and between the facilitator and participants. The interactionn tools include an emotions key that maps color to emotion, set of self-reflection questions, visual feedback (progress bar, step completion recognition, visual response), and audio response.

Training the model using Teachable Machine.

Integrating the participants' model in p5.js for color detection.

Setting up a virtual camera using OBS to broadcast the p5 web editor on Zoom.

06. Visual System

Once we trained our models and was able to integrate p5.js and OBS to trigger responses to the classes of images we registered, we dedicated to building an intuitive visual system for the interaction tools. We considered:

How many colors for what range of emotions? How specific should we map colors to emotions? What is an appropriate scope of the emotions? Can it be scaled?

For the visual feedback, we considered:

How to make it intuitive so that participants understand the visual responses to the completion of each activity step and emotional state? How to connect the visual responses to the activity of paper-folding, and make the two layers of visual representations (emotional states, paper-folding steps) visually cohesive?

We relied on existing color wheel to inform our color scheme and explored a visual language that was expressive, textured, composed of organic shapes, and playfull—all characteristics of paper cut-outs.

Visual exploration of various color schemes and shapes.

Mapping our color scheme and assets to emotions key.

10. Final Demo

Recording of a live zoom session.