The smartphone has become an important part of our daily lives. A 2022 report states that we spend an average of 4.8 hours a day on our mobile phones. In some markets it's even more than 5 hours! Another study from 2020 found that 88 percent of mobile internet usage is spent in apps. Exact numbers aside, it's clear that mobile app usage is high. What's more, many mobile users spend most of that time using only a limited number of apps. That's why it's important to ensure that your application is a pleasure to use and helps users easily achieve the required results. This is where mobile usability testing comes in. 

Mobile app usability testing is testing whether a mobile application helps users achieve their goals easily. It's often done by testers with knowledge of usability and design in conjunction with actual or potential users. It occurs when there's a user interface with some functionality for users to work with. This can be the user interface of the mobile app, but it could also be an interactive mockup.

Benefits of Mobile Usability Testing

Mobile usability testing provides several benefits. Some benefits depend on how the tests are conducted, but let's first cover these benefits. 

Finding Areas of Improvement

Usability tests can help you identify usability issues in the mobile application where improvement is still possible. Based on user feedback or metrics gathered during the tests, testers can discover which parts of the mobile app are difficult to work with for users, which features they can't find, or which tasks users have trouble completing. 

Without usability testing, these things may never be discovered. Aspects of the app may be logical to developers and testers because of their acquired knowledge. But users will be new to the app or even the problem domain.

Discovering Possible New Features

Mobile usability testing has several ways of discovering possible new features that may increase the value of your app. Looking closely at how users interact with your mobile application may tell you something about which features could improve their experience. Or you may serve test users questionnaires that they might use to share ideas for new features. If you never do any usability testing, you will either have to guess what to add to your app or hope your users are willing to contact you.

Finding Bugs

Every app development team does its best to write software without bugs. Automated tests are a great way to catch bugs early. But real app usage will always uncover new bugs. Usability testing can help teams uncover them. 

This is even more useful in the mobile app space because it may take a while for a new version of the app (with the bug fix) to make its way to the end users. 

Support for Previously Made Decisions

We face many choices when developing software. We have to choose which features to implement. Then we have to choose how to implement them. Good UX design is hard and can lead to many discussions among the team. It can be difficult to verify if a subsequent decision played out well or not. 

Mobile usability testing can provide an answer. Without usability testing, it's a lot harder to know the value of those UX decisions. And bad UX decisions can also mean a big drop in app usage.

Increased Engagement and Company Reputation

Users who enjoy working with an app will use it more, and they'll think positively about the company that created it. A company that doesn't conduct mobile usability tests will more or less rely on luck to increase user engagement or to bolster its public image. UX designs that seemed good during the design stage may not work so well for users in real life. Without usability tests, the company may only know about this when it's too late or after too much time, effort, and money have been invested.

Considerations for Usability Testing

Usability testing comes in many shapes and sizes. We can distill three aspects of usability tests that we need to consider. 

Remote or In-Person

In the past, usability tests were quite resource intensive. They required a test lab where the tester and the user could meet in real life. This lab would be a controlled environment where the user could interact with the application and the tester could observe the user interaction. Later, usability testing tools became available. 

For example, screen recording could record all user interactions on the mobile device in real time so that the tester can analyze the user behavior afterward. Nowadays, you can even track facial expressions and eye movement. Of course, this requires a well-equipped and sometimes costly lab. The company must invest in hardware, usability tools, one or more rooms for the tests to take place, good lighting, etc. Today, technology gives companies the option of remote usability testing. It can lower the barrier for many companies because there is no longer the need for a costly lab environment. 

Some tools can even save time as multiple users can do the tests simultaneously. Remote usability testing also lowers the barrier for users. They don't have to make the trip to the lab, and they can do the tests when it fits in their schedule. As such, companies can find more users with less effort. If the company is looking for quantitative data (i.e., it needs many users to do the tests), remote usability testing is a good choice.

Exploratory or Scripted

Exploratory usability testing is ideal for the early stages of development. It can even be done with mockups on paper. Exploratory testing provides qualitative data on design choices before you fully implement them. 

Ideally, they are done in person because the app is incomplete or doesn't even exist yet. A tester may need to explain certain aspects of the design and take note of the feedback. Scripted testing is done by giving the user a set of tasks they have to fulfill or steps they have to go through. The test then observes how users react to the application and how they execute the required steps or tasks. Scripted testing is a better fit for post-implementation tests and can be done remotely or in person. 

Moderated or Unmoderated

The last aspect to consider is the amount of moderation that's required. Both remote and in-person tests can be moderated or unmoderated. Exploratory testing sessions will often be more moderated than not, but scripted usability tests can be both. In moderated usability tests, the tester will actively participate. They can interrupt the user to ask how they feel or why they took certain actions. 

This is useful qualitative data but requires more time and money. Unmoderated tests are geared toward quantitative data. In these tests, users are observed without input from a moderator. Unmoderated usability tests can still include a set of instructions before the test and an evaluation form afterward. But the user is left to do the test on their own, and you can't always be sure the user understood the instructions correctly or that they really took the time to complete the evaluation.

Creating a Usability Test Plan

To conduct a usability test correctly, testers must work carefully and precisely in order to collect accurate data. Usability testing is akin to social science or psychology. A detailed and organized test plan is necessary. Without it, the testing process might not be done correctly, leaving the tester with unreliable results.

Outline Purpose and Goals

The first step toward a good usability test plan is to define the purpose and the goals of the test. These must be specific, well defined, and measurable. A good guideline is the SMART acronym. The goals must be: 

• Specific—they must target a specific feature or area of the app
• Measurable—the goals have to be quantifiable, i.e., there is a number that indicates progress toward the goal
• Assignable—the goal can be assigned to someone
• Realistic—taking into account time and resources, it's realistic to achieve the goals
• Time-related—you can define when the goals can be achieved

Formulate Research Questions

The next step is related to the previous step. Here, testers should formulate what they plan to test. They must come up with a clear definition that all parties agree on. In this step, testers can also formulate a hypothesis. 

Create a Task List

Now that the goals are defined and the research team knows what they want to test, they can move on to creating a task list. In this list, they'll outline the different steps they want the user to perform. Ideally, each task can be executed by the user without any help from the tester. They should also form a cohesive flow with the other tasks in the list. And of course, they must be tied to one or more of the previously outlined test objectives. 

Select Participant Characteristics

How to select test participants depends on your app and your goals. You want an accurate representation of the target audience of your app (or a subset). The selection can be based on a wide range of demographic or behavioral characteristics.

For example, a bank may want to test its app with users who are visually impaired. Or a company aiming to move to a new market may want to test with users in that country. It's important to notify your users that they'll be tested and have them sign a consent form before participating. This is especially true for minors, in which case you may also need consent from a parent or legal guardian. You should also outline on the form what will happen with the collected data.

Develop a Testing Method

In this step, testers should choose one of the aforementioned methodologies of usability testing: remote or in-person, exploratory or scripted, and moderated or unmoderated. This will determine when, where, and how the users will be tested. It will also determine if any tools should be acquired: rooms, recording devices, software, hardware, or anything else. In many cases, usability testers will write up a detailed testing procedure, including a time schedule. For example, for an in-person test, this could include the following parts: 

• Introduction—introducing the team, the app, and the goal of the test
• Pre-test interview—gathering data about the user before the user interacts with the app
• Test tasks—the user steps through the test
• Post-test interview—gathering data about the user after the user has interacted with the app
• Debrief—closing remarks

Having a well-defined procedure ensures that each test follows the same script and that data is comparable and valid.

Plan Reporting Process

Before conducting the tests, it's important to establish what insights the test hopes to provide (as mentioned above) and to define how the results of the test will be measured and reported to stakeholders. Again, this depends on the type of test. One big distinction to make is the difference between quantitative and qualitative data. Quantitative data can be expressed in numbers and is often easier to report but might lack nuance. The statistics from the beginning of this article are a good example of quantitative data (i.e., 88 percent of mobile usage is spent in apps). Qualitative data, on the other hand, is descriptive. It's about things we can observe but can't reduce to mere numbers.

Often, such reports contain longer pieces of text describing the results of the test. Qualitative tests often focus on a smaller group of users, while quantitative tests benefit from a larger pool of participants. It's important to know how you'll be reporting the test results, because without good and clear reporting, it will be difficult to define any next steps. Good reporting leads to actionable tasks for the team, while bad reporting leads to confusion about what to do next.

Tips for Mobile Usability Testing

Let's look at some useful tips for anyone aiming to conduct mobile usability testing. 

Carefully Select the Test Environment

The environment you test in will impact how your users interact with the app and therefore the results. Your app will be used in different environments in real life: at home, at the airport, in loud or silent places, in busy or calm places, and in places with good or bad reception. Take these factors into account when conducting usability tests.

Choose the Correct Testing Group and Size

The type of test you want to run will determine the correct testing group and size to work with. The demographics of the users and whether you want to do qualitative or quantitative research, for example. If you want a concrete number, Jakob Nielsen tells us that you can gather significant qualitative data from only five users and quantitative data from twenty.

Don't Rely on a Prototype

We mentioned that it's possible to perform usability testing on paper or with a working prototype. But if you can, provide users with a real app on a real device (and different devices at that). This will provide the most reliable results from your tests.

Keep Testing Short and Succinct

Make sure users understand what's expected of them, and don't drag them along for longer than half an hour. Any longer and they might lose interest, and your results will not be representative of how real users would interact with the app.

Conclusion

Usability testing is a research method to see how users react to your app and how or if they can successfully complete a given set of tasks. Mobile usability testing focuses on mobile apps. There are several benefits of mobile usability, as you saw. The overall advantage is that it can help you produce a more useful app with a better user experience, happier users, and a better reputation for the company. 

There are several ways of conducting mobile usability testing that testers should take into consideration. These also have influence on the choices testers should make when creating a usability test plan. This plan should include clear goals and clear steps for users to take. The testers should also think well about who they want to include in the test. While Waldo isn't a pure usability testing tool, there are certain aspects that can provide initial insights into the usability of your mobile app.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.