Duolingo usability study header image

Duolingo

Usability Study

Duolingo is a digital language learning tool that allows users to practice dozens of languages, accessible both through mobile and the web. We conducted usability tests on the Japanese and Korean “learning a new writing system” tools and provided design recommendations to improve users’ experience with learning unfamiliar writing systems. This case study is a non-sponsored, team-based project I worked on in the Usability Study course at the University of Washington.

My Role & Contributions

UX Designer, Design Researcher

  • Crafted research protocol

  • Facilitated usability testing sessions

  • Ran data analysis (affinity mapping)

  • Provided design recommendations and visual design

Timeline

Jan 2022 - Mar 2022 (10 weeks)

Team

3 UX Researchers

Tools

Figma
Miro
Zoom

CONTEXT

Help English speakers better navigate new writing systems

Unfamiliarity with non-Latin based language

Non-Latin-based languages are generally quite difficult to learn for English speakers accustomed to the Latin alphabet due to its unfamiliar writing systems, grammar, etc. Duolingo also occasionally suffers from inconsistencies in volunteer resources wherein “popular” Latin-based languages have better or more thoroughly workshopped features.

Character Learning

Character learning is the essential first step for non-native speakers to familiarize themselves with new writing systems. We decided to explore how users interact with this character feature and discover which usability positivities and shortcomings currently exist on its platform.

Duolingo writing system lesson interface

Methods and Procedures

Language

Japanese & Korean

Interface

Web interface

Testing Types

Moderated / Remote

Platform

Zoom

Research Questions

  • How easily and successfully do users use Duolingo's "learn a new writing system" tool?
  • What worked well for users while learning a new writing system?
  • What obstacles or frustrations did users experience while learning a new writing system?

Target Audience

After collecting the data from our screener questionnaire, based on people's interests, we decided to target people who meet the following requirements. We recruited 10 participants (5 for Japanese and 5 for Korean studies) from the screener respondents.

  • People who want to learn either Japanese or Korean
  • People who are new to the respective writing systems (cannot read or write it)
  • People who are new to Duolingo or have limited prior experience

Testing Tasks

We asked the participants to complete 3 main tasks thinking aloud to understand their holistic experience with the character-learning tool.

1. Home page exploration

Home page screen

2. Character page exploration

character page screen

3. Complete character lessons (2 rounds)

character lesson screen

Data Analysis

We moved the findings to a Miro board and created Affinity Diagrams to consolidate and categorize large data. We discussed the scope and severity of each theme and determined what needs to be prioritized.

Affinity diagrams image

Key Findings & Recommendations

After discussing the design recommendations with the team, I designed the visual examples of our proposed solutions.

1. Current wrong answer mechanisms causes low learnability

Scope: Mid

Severity: High

  • Participants didn't notice the correct answer displayed in the bottom bar.
  • Participants wanted to go back and recheck the right answer, but they couldn't.
  • Participants didn't realize the questions they got wrong were repeated during the quiz and made the same mistakes.

Those frictions interfere with users' knowledge acquisition. The learnability of the design could be improved by addressing the issues.

Character lesson screen
Proposed Solution
  • Directly display the correct answer in the quiz instead of showing it in the bottom bar. Looking at the character and its correct sound side by side would enhance users' memory.
  • Communicating that the same question will be repeated to users would encourage them to remember the correct answer.
proposed solution image

2. Lack of Instruction made users guess what to do (Pair Matching Exercise)

Scope: Mid

Severity: Mid

Participants couldn't quickly tell if they were supposed to match the items on the left column with those on the right column. Some also assumed they would drag the buttons instead of clicking.
This indicated that the design was not intuitive and lacked instructions.

"I don't know how we're supposed to match them because there are no arrows or anything so I'm not sure if we're supposed to drag or directly pair them up"
Pair matching exercise screen
Proposed Solution
  • Provide a more intuitive design by making a clearer distinction between the left and right columns and showing a line connecting the selected items
  • Provide visual instructions with animations to show users what to do
proposed solution image

3. Lack of Instruction made users guess what to do (Build Character Exercise)

Scope: Mid

Severity: Mid

Participants had to figure out how to build a character by randomly clicking the buttons since they didn't understand how exactly Korean characters were built yet. While they were able to learn it quickly, it was not an intuitive and pleasant experience.

“There were no cues for this exercise and you just clicked on the buttons to put them in the right box, I thought you had to trace them”
Pair matching exercise screen
Proposed Solution
  • Adding a more thorough explanation that Korean characters are made of building blocks and visual examples enhance assurance and learnability
  • Include an animation of a few characters being build
Proposed solution image

4. Users want a clear structure of the study plan

Scope: low

Severity: high

In each lesson, several characters were randomly selected, and there were no clear explanations of the relationship among the chosen characters. Participants felt unsure how the lessons play a part in learning the language overall.

“I wish it would have been a little more upfront on what exactly and how exactly I'm learning, I wanted to see a clear lesson plan.”
character lesson screen

Randomly selected characters (ko, shi, su, u)

Proposed Solution
  • The five vowels (a, i, u, e, o) are combined with the consonant sounds (k, s, t, n, h, m, y, r, w) to produce almost all the sounds represented by Japanese characters. Therefore, introducing characters with the same consonant would make more sense to the users than randomly selected characters.
proposed solution image
  • Show some examples of words they can make from the characters they learned at the end of the lesson.
proposed solution image

5. Unexpected Audio Play

Scope: High

Severity: Low

  • Sound plays immediately after a lesson starts without notice, which is unexpected for users
  • Participants want to know that the lesson would have the sound effect before starting the lesson
Sound icon image
Proposed Solution
  • Notify users of the audio feature and allow users to confirm it before starting the first lesson
Recommendation image

Study Improvements

Recruit more diverse participants to validate our findings

  • Most of our participants had some level of experience learning other foreign languages
  • Demographics of the participants were not diverse enough (mostly students, quite a few UW students)
  • We want more feedback from people who are not familiar with non-Latin writing systems at all

Learnings

Scoping is crucial for a usability study

Originally, we wanted to recruit people who were interested in learning any new language with an unfamiliar writing system, but narrowing the scope down to 2 specific languages made our project manageable and helped us identify what to focus on the most during the testing.

Know the difference between simple and intuitive design

While the participants liked Duolingo's simple and clean interface, I found that making the design too simple could make it less intuitive. I believe designers should identify the sweet spot that enables both simplicity and intuitive design.

Anise Health

Microsoft

NWESS

Premera Blue Cross

Social Impact Board

Duolingo

Let's connect,

if you want to learn more about my projects, give feedback, or work on an exciting project together!

©2023 | TOMOMI MATSUZAKI