Throughout the Waldo on Waldo (WoW) project, our ultimate goal has been to shift our testing strategy as far left as possible. We aimed to achieve a mandatory Waldo status on our pull requests, ensuring that changes to our web app could only be merged after passing the WoW tests.

To accomplish this without creating a bottleneck, it was crucial for our test suite to run in under 30 minutes (ideally under 20) with a concurrency of 10 devices. In this article, we'll explore the performance and cost optimizations we implemented to achieve these objectives.

Streamlining Signups

In our initial setup, we had Waldo fill in all the fields in the signup form for each test run. However, this approach proved suboptimal and time-consuming, as the fields remained the same for every test, except for the email address. Instead, we implemented a “dynamic login” to generate a new account on the fly for any email not yet on our system. Once again, this behavior is only available in our staging environment. This simple change, implemented by updating our login backend logic, reduced the number of interactions by 30% across our 65 tests, significantly improving overall execution duration.

Bypassing Test Replay Backend Logic

Another optimization specific to our product involved bypassing our test replay backend logic. In certain tests, we needed to relaunch Waldo Replays after updating a step interaction. Normally, this process would generate a new Replay in our backend, which could take anywhere from 2 to 5 minutes. To avoid this delay, we decided to bypass the backend logic for WoW Replays. Instead, we immediately copied the previous Replay. This modification resulted in a significant reduction in test duration, with some tests experiencing a time decrease from 6 minutes to just 1 minute.

Improved Device Configuration Management

We made significant improvements to our device configuration management using the Rules feature. Initially, our full test suite ran on both a regular iPad and a wide iPad for responsiveness validation. However, we found that conducting responsiveness validation for every pull request (PR) was excessive.

To optimize our testing process, we created separate rules for PRs and the main branch. In other words, any pull request in GitHub will trigger the full test suite on 1 device, and once that pull request is merged to the main branch, we run that same test suite on 2 devices. This ensures that responsiveness validation is performed exclusively on the main branch, reducing unnecessary testing overhead for PRs.

‍

Results and Impact

Collectively, these enhancements allowed us to achieve our target of completing the test suite within the 30-minute limit (15 to 20 minutes on average), with a total of approximately 3,500 interactions per test suite per device. As shown in our monthly usage graph, we perform an average of 300,000 interactions per month. This translates to 300,000 potential opportunities for identifying and addressing bugs before PRs are even reviewed.

The impact of WoW has been transformative for our team. It empowers us to consistently ship cleaner code, forces us to dog-food our own product, and ultimately ensures a higher level of quality and reliability.

In conclusion, our focus on performance and cost optimizations has elevated the WoW project to new heights, enabling us to uncover potential issues early in the development process and maintain a seamless and efficient testing workflow. It also allows us to see the pain of our customers and see where we (and therefore they) suffer the most. We have been able to suggest most of these techniques to our customers!

Empowered Testing with Waldo on Waldo (WoW)

Throughout this four-part article series, we have explored the transformative power of Waldo on Waldo (WoW) in our web app testing process. WoW has allowed us to leverage our own testing solution to ensure the quality, reliability, and efficiency of our web app. Let's recap the key insights from each part:

In Part 1: Welcome to Waldo on Waldo we explored the concept of dogfooding and highlighted the significance of using our own solution to test and improve our web app. By overcoming platform limitations and addressing keyboard issues, we integrated our web app into Waldo, enabling us to experience and verify it as our users would.

Part 2: User State Management delved into the importance of managing user states in testing. We explored the two approaches we considered and implemented fresh user accounts, streamlining the user plan upgrade within WoW. By automating user state management, we ensured consistent and reliable testing results.

In Part 3: Clone Data for Complex Scenarios, we delved into the complexities of creating and managing data for various testing scenarios. Implementing data cloning endpoints in our backend allowed us to streamline the process and test a wide range of use cases. With data cloning, we achieved greater flexibility and efficiency in our testing efforts.

Finally, Part 4: Performance and Cost Optimizations revealed our pursuit of efficient and cost-effective testing. Through optimizations such as streamlined sign-up processes, bypassing unnecessary backend logic, and improved device configuration management, we were able to reduce test execution times and infrastructure costs.

Waldo on Waldo has revolutionized our testing process, enabling us to ship cleaner code more consistently and discover potential bugs before they reach our customers. By dogfooding our own solution, we have not only improved our web app but also demonstrated our commitment to the quality and reliability of the tools we provide.

We hope that our journey with WoW has provided insights and inspiration for your own testing endeavors. Embrace the power of dogfooding, address testing challenges head-on, and continuously seek opportunities for optimization. Together, we can build robust and user-centric applications. Happy testing!

‍