Mobile Kombat: Manual Testing vs Waldo

You have too many mobile tests to run…and it’s time to choose your fighter: hire more manual QA testers, or take the leap and invest in test automation. 

For many mobile teams, this decision first presents itself when you have reached a point in your mobile app’s development where quality becomes a topline priority. Not finding market fit, simply continuing to refine and patch your existing app to keep users happy. 

This critical inflection point can happen very early in your app’s journey, especially if you are in an industry where bugs don’t just cause unpleasant user experiences: they can cause real issues. If an AllTrails user is suddenly locked out of their account 3 days into an off-trail hike, or a Rocket Money user has their auto-pay function compromised…the impact of those bugs extends into the day to day lives of the end users. 

Of course this is not a choice between manual testing OR automation. Manual testing is an extremely important part of your mobile testing coverage. Edge cases, general usability testing, or just ad hoc tests you want to run in response to bug tickets, etc. are perfectly suited for manual testing processes. 

It is a choice between when you should continue to invest in headcount dedicated to manual testing to achieve your desired coverage across other test cases (regression testing, smoke tests, etc.), or start putting that money towards automation. 

‍

Ramping up from 0 dedicated QA hires

Let’s say you have no dedicated QA employees: your current testing needs are covered by your development team, product managers, maybe even customer success. You reach a point where the testing requirements to maintain your desired coverage are too much for these team members to juggle with their other responsibilities. 

The next natural step here is to hire your first, dedicated, QA employee. Having an employee who’s single focus is testing can increase coverage while reducing the amount of time every other team member is asked to devote to testing.

Waldo has successful customers come on board with 0 dedicated QAs, such as Opus. The team is leveraging a more moderate implementation: with engineering maintaining a set of 10-20 regression test scenarios per platform, across multiple device types. 

While this allowed them to improve the quality of their test coverage while reducing the amount of time the developers had to spend on testing, taking this same approach would not work well across a more comprehensive suite of 50+ flows. At that point, you need to have someone who is focused on managing testing update Waldo to ensure you do not devolve into cannibalizing engineering time. 

‍

Teams with a single dedicated QA

Teams that have a single dedicated mobile testing resource often immediately feel pressure to add a second. If quality is truly a priority: that single employee will constantly be asked to test more scenarios, increase device coverage, move faster, and deliver high quality results to the engineering team.

At this inflection point, teams often hire a second dedicated tester instead of exploring automation, since traditional, scripted, mobile test automation requires a lot of up front (and long tail) investment for ultimately unreliable results. 

For most teams, Waldo changes this calculation because a single dedicated QA hire can manage an entire automated mobile end-to-end test suite. For example: instead of building a team of people out around their first hire, Konfio elected to empower their dedicated QA lead with Waldo. Their team of 1 was able to create and maintain 100+ test scenarios run daily, and verify their new releases in just 30 minutes. If anything in the suite breaks, the engineering team is alerted, quickly makes required changes, and re-runs the suite before release to ensure everything is passing. 

‍

If you have more than 3 team members dedicated to QA

In our experience, teams that have 3+ hires that are only focused on manual QA for the mobile side of the business have invested in manual QA because of a poor experience with scripting. They reached the inflection point above where they considered a second QA hire, they brought on a QA automation engineer, and then after months of hard work: they found themselves with only a handful of functioning, non-flaky, scripted tests. 

Waving the white flag, they returned to relying on manual testing with their expanded headcount. However, maintaining a team of 3+ hires (that is still being overwhelmed with the ever growing testing demands) becomes overwhelmingly expensive without a commiserate return on effectiveness or quality.

‍

In conclusion...

Waldo makes it easier to implement mobile test automation, using current resources, by making it possible for technical or non-technical resources to build fully automated tests in minutes. 

Disclaimer: Waldo is not magic, and like all the other options above has its pros and cons. In order to deliver high value, Waldo requires proper setup and implementation which needs to be led by engineering or developers. However, the work required is basic automatability work that any automation solution would require to ensure functions of your app can be programmatically tested. 

The difference is that doing this work with Waldo results in a long term testing solution that can be maintained by technical or non-technical team members in a fraction of the time with existing resources. If you are exploring a sustainable, long term solution for scalable mobile testing: look no further than Waldo. 

To learn more about how Waldo can help your team start benefiting from mobile test automation, book a demo today!