Originally posted here on January 27th, 2021 by Pierre-Yves Ricau , Android tech lead at Square. Pierre-Yves, had his doubts on “script-less” testing platforms. After speaking with our CEO, Amine, he decided to take Waldo for a test run and was determined to break it… Did he succeed? Read on to find out.
Today, Waldo.io, a script-less platform for automated mobile tests, announced general availability of their Android support.
I was recently talking with the CEO of Waldo, Amine Bellakrid. I told him that the llama logo was super cute and the UI gorgeous… but I had doubts a no-code platform was a good fit for automated mobile tests.
The problem with no-code tests
No-code platforms typically provide a recorder tool that lets you navigate through the app, recording everywhere you tapped. Then later on you can replay the test automatically, and get notified when it fails.
Automated tests are expected to fail when a change breaks the correct behavior of the app. Unfortunately, no-code platforms tend to generate brittle tests that also fail on small changes that didn’t introduce incorrect behavior.
This creates a lot of noise, so teams tend to stop updating and running these tests and they just rot in a corner.
Note: similar problems can happen with code based tests, but we’ve introduced patterns to work around them (e.g. Testing Robots)
Apparently Waldo is different
When I shared my doubts, Amine smiled and said I should try Waldo. So I created a new example project with 3 tabs:
I recorded a simple test on Waldo.io, tapping on the second tab (Dashboard) and then the 3rd tab (Notifications).
Messing with Waldo
Tab swap
I started by swapping out the second and the third navigation tab:
The result is interesting: Waldo clicked on the correct tab (Dashboard) even though it changed place, and the test passed.
Changing strings
Next, I renamed the Dashboard tab to Summary . Still ✅.
Changing everything else
Ok, time to make this really hard. I also changed the tab menu id and its icon. No way this can still work.
… the test still passes and finds the right tabs to tap on, in the right order.
One thing that’s really cool in the screenshot above:
Screen similarity should be at least 70%.
Current similarity: 93%
I don’t know how the similarity is calculated, but it seems more advanced than a simple matching on view ids, bitmap content or string content. That explains why Waldo is totally fine with me changing the UI.
Ok but I really want to break it
I decided to just be mean and remove the tab. Take that Waldo!
This time the test is failing
.
Let’s see what happens when I click Update test:
The recorder is on the right, the new scenario at the top left and the old scenario at the bottom left. Now that I have only two tabs, I click on the Notifications tab and the scenario is immediately updated. Nice!
Conclusion
I changed my mind: no-code test platforms don’t have to be brittle!
I don’t know all the details of how Waldo does it, so I hope they can share more about it some day and bring the community forward. In the meantime, I would definitely give Waldo a shot.
Full disclosure: Amine is my friend and I’ve been advising the Waldo team every now and then on some cool Android tech problems. I wrote this blog on my own though.
Automated E2E tests for your mobile app
Get true E2E testing in minutes, not months.
