Duolingo usability study header image

Duolingo Usability Study

Duolingo is a digital language learning tool that allows users to practice dozens of languages, accessible both through mobile and the web. We conducted usability tests on the Japanese and Korean “learning a new writing system” tools and provided design recommendations to improve users’ experience with learning unfamiliar writing systems.

This case study is a non-sponsored, team-based project I worked on in the Usability Study course at the University of Washington.

My Role

UX Designer, Design Researcher

Moderated Usability Study, Interface Redesign

Team

Camille Bourbonnais, Naomi Johnson, Yimeng Wang, Ruican Zhong

Timeline

Jan 2022 - Mar 2022

Tools

Figma, Miro, Zoom

The Problem

Unfamiliarity with non-Latin based language

Non-Latin based languages are generally quite difficult to learn for English speakers accustomed to the Latin alphabet due to its unfamiliar writing systems, grammar, etc. Duolingo also occasionally suffers from inconsistencies in volunteer resources wherein “popular” latin-based languages have better or more thoroughly workshopped features. Our team was interested in examining how their perceived “unfamiliarity” shapes the overall language learning experience on Duolingo, as well as discovering which usability positivities and shortcomings currently exist on its platform.

Country flags

Define the Scope

The Writing Systems

8 non-Latin based languages have a specific character-learning tool, shown separately from the main lesson tree on Duolingo. As we saw this tool as the essential first step for non-native speakers to familiarize themselves with the language’s respective writing system, we decided to explore how users interact with this character feature to properly explore their experiences with learning unfamiliar writing systems.

Our Goal

By providing design recommendations for Duolingo’s “character-learning” tool, we help English speakers better navigate new writing systems.

Duolingo writing system lesson interface

Methods and Procedures

Language

Japanese & Korean

Interface

Web interface

Testing Types

Moderated / Remote

Platform

Zoom

Research Questions

Target Audience

After collecting the data from our screener questionnaire, based on people's interests, we decided to target people who meet the following requirements. We recruited 10 participants (5 for Japanese and 5 for Korean studies) from the screener respondents.

Target Audience

People who want to learn either Japanese or Korean

    Target audience

    People who are new to the respective writing systems

    (Cannot read or write it)

    Target audience

    People who are new to Duolingo, or have limited prior experience

    (If the participants have used Duolingo before, they should have only used it a few times in the past year.)

    Testing Tasks

    We asked the participants to complete 3 main tasks thinking aloud to understand their holistic experience with the character-learning tool.

    Home page screen

    1. Home page exploration

    Navigating Duolingo's homepage and explaining what it communicates to them

      character page screen

      2. Character page exploration

      Navigating Duolingo's characters page and explaining what it communicate to them

      character lesson screen

      3. Complete characters lessons

      Completing two characters lessons while thinking aloud to measure usability and learnability

      Data Analysis

      We move the data to a Miro board and created Affinity Diagrams to map out and consolidate a large amount of information. We discussed the scope and severity of each finding and determined what needs to be prioritized.

      Affinity diagrams image

      Key Findings & Recommendations

      I took a lead and designed the vial examples of our proposed solutions.

      1. Current wrong answer mechanisms causes low learnability

      Scope: Mid

      Severity: High

      • Participants did not notice it was displaying the correct answer in the bottom bar.
      • Participants tried to go back and check the right answer again, but couldn't.
      • Participants took some time to realize the questions they got wrong were repeated and made the same mistakes.

      We believe low awareness of their own mistakes could interfere with users' acquisition of new knowledge. Thus, the learnability of the design is low.

      "When I get it wrong, I don't really notice what I got wrong."
      Character lesson screen
      Proposed Solution
      • Directly display the correct answer in the quiz. Looking at the character and the correct sound side by side would enhance their memory.
      • Communicating that the same question will be repeated to users would encourage them to remember the correct answer
      proposed solution image

      2. Lack of Instruction made users guess what to do (Pair Matching Exercise)

      Scope: Mid

      Severity: Mid

      It was not very clear for participants that they were matching the sounds on the left with the characters on the right column. Some also assumed they would drag the buttons instead of clicking. This happened because the design was not intuitive and lacked instructions.

      "I don't know how we're supposed to match them because there are no arrows or anything so I'm not sure if we're supposed to drag or directly pair them up"
      Pair matching exercise screen
      Proposed Solution
      • More intuitive design: Make a clearer distinction between the left and right columns. Show a line connecting the selected two buttons
      • More instructions: Use animation to show users what to do
      proposed solution image

      3. Lack of Instruction made users guess what to do (Build Character Exercise)

      Scope: Mid

      Severity: Mid

      Participants had to figure out how to build a character by randomly clicking the buttons since they didn't understand how exactly Korean characters were built yet. While they were able to learn it quickly, it was not an intuitive and pleasant experience.

      “There were no cues for this exercise and you just clicked on the buttons to put them in the right box, I thought you had to trace them”
      Pair matching exercise screen
      Proposed Solution
      • Adding a more thorough explanation that Korean characters are made of building blocks and visual examples enhance assurance and learnability
      • Include an animation of a few characters being build
      Proposed solution image

      4. Users want a clear structure of the study plan

      Scope: low

      Severity: high

      Several characters were randomly selected for each lesson with no clear instructions of what they were learning. Participants felt unsure how the lessons play a part in learning the language overall.

      “I wish it would have been a little more upfront on what exactly and how exactly I'm learning, I wanted to see a clear lesson plan.”
      character lesson screen

      Randomly selected characters (ko, shi, su, u)

      Proposed Solution
      • Five vowels (a, i, u, e, o) are combined with the consonant sounds (k, s, t, n, h, m, y, r, w) to produce almost all the sounds represented by Japanese characters. Therefore, we think introducing characters that have the same consonant together would make more sense to the users than randomly selected characters.
      proposed solution image
      • Show some example words that they can make out of the characters they learned after the lesson.
      proposed solution image

      5. Unexpected Audio Play

      Scope: High

      Severity: Low

      • Sound plays immediately after a lesson starts without notice, which is unexpected for users
      • Participants want to know that the lesson would have the sound effect before starting the lesson
      Sound icon image
      Proposed Solution
      • Add a notification about the sound and allow users to confirm it before starting the first lesson
      Recommendation image

      Study Improvements

      Recruit more diverse participants to validate our findings

      • Most of our participants had some level of experience learning other foreign languages
      • Demographics of the participants were not diverse enough (mostly students, quite a few UW students)
      • We want more feedback from people who are not familiar with non-Latin writing systems at all

      No need to keep track of time

      We realized the time each spent on a lesson was not a useful usability metric in our study, as as it takes longer when the participants think aloud and the facilitator asked questions. We ended up focusing more on qualitative data.

      Learnings

      Scoping is crucial for a usability study

      Originally, we wanted to recruit people who were interested in learning any new language with an unfamiliar writing system, but narrowing the scope down to 2 specific languages made our project manageable and helped us identify what to focus on the most during the testing.

      Know the difference between simple and intuitive design

      While the participants liked Duolingo's simple and clean interface, I found that making the design too simple could make it less intuitive. I believe designers should identify the sweet spot that enables both simplicity and intuitive design.

      Next Project >

      Let's connect