Duolingo

Usability Study

Duolingo is a digital language learning tool that allows users to practice dozens of languages, accessible both through mobile and the web. We conducted usability tests on the Japanese and Korean “learning a new writing system” tools and provided design recommendations to improve users’ experience with learning unfamiliar writing systems. This case study is a non-sponsored, team-based project I worked on in the Usability Study course at the University of Washington.

My Role & Contributions

UX Designer, Design Researcher

Crafted research protocol
Facilitated usability testing sessions
Ran data analysis (affinity mapping)
Provided design recommendations and visual design

Timeline

Jan 2022 - Mar 2022 (10 weeks)

Team

3 UX Researchers

Tools

Figma
Miro
Zoom

Methods and Procedures

Language

Japanese & Korean

Interface

Web interface

Testing Types

Moderated / Remote

Platform

Zoom

Research Questions

How easily and successfully do users use Duolingo's "learn a new writing system" tool?
What worked well for users while learning a new writing system?
What obstacles or frustrations did users experience while learning a new writing system?

Target Audience

After collecting the data from our screener questionnaire, based on people's interests, we decided to target people who meet the following requirements. We recruited 10 participants (5 for Japanese and 5 for Korean studies) from the screener respondents.

People who want to learn either Japanese or Korean
People who are new to the respective writing systems (cannot read or write it)
People who are new to Duolingo or have limited prior experience

Testing Tasks

We asked the participants to complete 3 main tasks thinking aloud to understand their holistic experience with the character-learning tool.

1. Home page exploration

2. Character page exploration

3. Complete character lessons (2 rounds)

Data Analysis

We moved the findings to a Miro board and created Affinity Diagrams to consolidate and categorize large data. We discussed the scope and severity of each theme and determined what needs to be prioritized.

Key Findings & Recommendations

After discussing the design recommendations with the team, I designed the visual examples of our proposed solutions.

1. Current wrong answer mechanisms causes low learnability

Scope: Mid

Severity: High

Participants didn't notice the correct answer displayed in the bottom bar.
Participants wanted to go back and recheck the right answer, but they couldn't.
Participants didn't realize the questions they got wrong were repeated during the quiz and made the same mistakes.

Those frictions interfere with users' knowledge acquisition. The learnability of the design could be improved by addressing the issues.

Proposed Solution

Directly display the correct answer in the quiz instead of showing it in the bottom bar. Looking at the character and its correct sound side by side would enhance users' memory.
Communicating that the same question will be repeated to users would encourage them to remember the correct answer.

2. Lack of Instruction made users guess what to do (Pair Matching Exercise)

Scope: Mid

Severity: Mid

Participants couldn't quickly tell if they were supposed to match the items on the left column with those on the right column. Some also assumed they would drag the buttons instead of clicking.
This indicated that the design was not intuitive and lacked instructions.

"I don't know how we're supposed to match them because there are no arrows or anything so I'm not sure if we're supposed to drag or directly pair them up"

Proposed Solution

Provide a more intuitive design by making a clearer distinction between the left and right columns and showing a line connecting the selected items
Provide visual instructions with animations to show users what to do

3. Lack of Instruction made users guess what to do (Build Character Exercise)

Scope: Mid

Severity: Mid

Participants had to figure out how to build a character by randomly clicking the buttons since they didn't understand how exactly Korean characters were built yet. While they were able to learn it quickly, it was not an intuitive and pleasant experience.

“There were no cues for this exercise and you just clicked on the buttons to put them in the right box, I thought you had to trace them”

Proposed Solution

Adding a more thorough explanation that Korean characters are made of building blocks and visual examples enhance assurance and learnability
Include an animation of a few characters being build

4. Users want a clear structure of the study plan

Scope: low

Severity: high

In each lesson, several characters were randomly selected, and there were no clear explanations of the relationship among the chosen characters. Participants felt unsure how the lessons play a part in learning the language overall.

“I wish it would have been a little more upfront on what exactly and how exactly I'm learning, I wanted to see a clear lesson plan.”

Randomly selected characters (ko, shi, su, u)

Proposed Solution

The five vowels (a, i, u, e, o) are combined with the consonant sounds (k, s, t, n, h, m, y, r, w) to produce almost all the sounds represented by Japanese characters. Therefore, introducing characters with the same consonant would make more sense to the users than randomly selected characters.

Show some examples of words they can make from the characters they learned at the end of the lesson.

5. Unexpected Audio Play

Scope: High

Severity: Low

Sound plays immediately after a lesson starts without notice, which is unexpected for users
Participants want to know that the lesson would have the sound effect before starting the lesson

Proposed Solution

Notify users of the audio feature and allow users to confirm it before starting the first lesson

Study Improvements

Recruit more diverse participants to validate our findings

Most of our participants had some level of experience learning other foreign languages
Demographics of the participants were not diverse enough (mostly students, quite a few UW students)
We want more feedback from people who are not familiar with non-Latin writing systems at all

Learnings

Scoping is crucial for a usability study

Originally, we wanted to recruit people who were interested in learning any new language with an unfamiliar writing system, but narrowing the scope down to 2 specific languages made our project manageable and helped us identify what to focus on the most during the testing.

Know the difference between simple and intuitive design

While the participants liked Duolingo's simple and clean interface, I found that making the design too simple could make it less intuitive. I believe designers should identify the sweet spot that enables both simplicity and intuitive design.

Duolingo

Usability Study

My Role & Contributions

Timeline

Team

Tools

CONTEXT

Help English speakers better navigate new writing systems

Unfamiliarity with non-Latin based language

Character Learning

Methods and Procedures

Language

Interface

Testing Types

Platform

Research Questions

Target Audience

Testing Tasks

1. Home page exploration

2. Character page exploration

3. Complete character lessons (2 rounds)

Data Analysis

Key Findings & Recommendations

1. Current wrong answer mechanisms causes low learnability

Proposed Solution

2. Lack of Instruction made users guess what to do (Pair Matching Exercise)

Proposed Solution

3. Lack of Instruction made users guess what to do (Build Character Exercise)

Proposed Solution

4. Users want a clear structure of the study plan

Proposed Solution

5. Unexpected Audio Play

Proposed Solution

Study Improvements

Recruit more diverse participants to validate our findings

Learnings

Scoping is crucial for a usability study

Know the difference between simple and intuitive design

Anise Health

Microsoft

NWESS

Premera Blue Cross

Social Impact Board

Duolingo

Let's connect.