Mobile Kombat II: Scripting vs Waldo

It’s an inflection point every major mobile app team will reach: when the testing needs around every release leave you no choice but to invest in mobile test automation. 

Automated mobile testing simply scales where manual testing cannot: unlocking greater efficiency and increasing your testing coverage.

However, scripting demands massive investment in order to be successful and is difficult to implement correctly…especially for mobile. 

Many teams that say they have test automation in place only have a small fraction of their testing needs covered with automation. What they often have is a black hole of resources that is pulling developer time and attention away from their core responsibilities with the app itself.  

‍

Scripting is a "critical mass" problem

Since scripted tests are so prescriptive, and hard coded to certain attributes of a build they often produce flaky results that are hard for teams to trust. This is especially true if you are using a framework that does not support native tooling (React Native, Flutter, etc.). Flaky tests do not provide clear signals, and results from the test don’t always speak to the functionality of your build. This leads to teams learning to ignore the results, relying on manual checks to confirm outcomes, or never running the tests in the first place. 

Sure, your team is catching the glaringly obvious bugs that impact core user flows a majority of the time…but those bugs likely are not caught because of any test automation. That means developers spending time re-doing the work these automated tests were supposed to address, while still potentially hiring dedicated QA engineering (or SDETs) to run and maintain the automation you have in place. 

From writing the tests to maintaining them (and spinning up + maintaining the infrastructure), scripting becomes a critical mass problem for every team: if you throw enough money and resources at it, you will eventually find some level of success. The question is: can you afford to continue building to that point without the process becoming too expensive or unreliable? 

‍

What are my options? 

Many teams feel like they don’t have a choice: scripting is the only solution they know for test automation. Waldo offers an alternative to scripting for mobile teams that need to quickly unlock the benefits of automated testing, but don’t have the resources required to see success in the near term with scripting. 

To begin, you upload a build to the platform, select a simulated device in the configuration of your choosing. Waldo provides a fully maintained testing infrastructure for its users so you will never have to set up devices or update servers. 

Building a test in Waldo is as easy as using your app. After you have selected your device, begin to move through your test scenario. Waldo will capture all of your actions as you move from screen to screen and save this as an example of the ideal path for that user flow. 

When you upload a new build, Waldo attempts to recreate that same path through the test scenario. It will document any discrepancies it finds along the way, and fail your test if it is unable to navigate the flow. 

Once your test is complete, you can go into the test results and view: 

• A full video playback of your test 
• A timeline of every action taken in the test 
• Screen by screen breakdowns of how Waldo analyzed the build compared to the same step in the “ideal path”
• Flakiness warnings on the test itself
• Network requests and console logs

This ensures that it is very clear how Waldo analyzed each screen, any issues that it flagged for review and what caused it to fail or pass a test. You know the results are reliable, and a failed test clearly indicates an issue with your build.   

‍

A few things to note...

Because Waldo conducts testing based on replicating a “happy path” through your app, it has limitations. Certain testing, like edge cases or negative tests, are not as well suited (or always possible) in Waldo. 

With scripts, their greatest strength is that they are composed with code…but that is also their greatest weakness. Scripts are written by your team and can be tailored to include commands that fulfill your unique conditions, but that level of customization introduces additional maintenance work and greater potential for flakiness. 

The “happy path” method of testing allows Waldo to significantly reduce test maintenance work, but it also means that certain tests simply will not be possible in the tool. 

Implementing Waldo will help your team benefit from automated testing across a majority of your needs, using a fraction of the resources, with significantly reduced maintenance work.

Reach out today to book a demo.

‍