Introducing Pipeline Gatekeeper

For years, mobile development teams have lacked the necessary tools and frameworks to make mobile testing an accelerant in the software development lifecycle. Today, I am excited to announce a paradigm shift for mobile development and testing. With the announcement of our latest release, which includes three new features, Pipeline Gatekeeper, Rules, and Branches, mobile teams can finally implement a smarter strategy for mobile testing by shifting testing left.

With Pipeline Gatekeeper, you can now instruct Waldo to run specific tests on PRs. Being able to run Waldo on PRs has numerous benefits, including:

• Faster release cycles because bugs caught earlier in the cycle are fixed faster
• Easy identification of what code changes caused a test failure
• The quality of the builds that go to your QA phase are far higher
• Fewer late cycle delays due to better testing on PRs and main branches
• Confidence that critical flows are tested constantly with test prioritization

With Rules and Branches, development teams can now:

1. Run the right test suites at the right times (e.g., on PRs, nightly builds, pre-release, etc)
2. Update the tests on your feature branch without impacting the tests run on your main branch
3. Share your updated tests across the team once your feature branch is merged
4. Merge your test suite in parallel with your code so updated tests can be shared across the team

To improve the mobile testing process, technology is only part of the solution. A mindset shift from engineering teams and leadership is also necessary. We have three core principles that will unlock a continuous testing process for mobile development teams.

1. Testing needs to be a shared responsibility
2. Testing needs to be efficient
3. Testing needs be transparent

Testing needs to be a shared responsibility

In too many organizations, there is a constant struggle between engineers and QA over who is responsible for creating and maintaining tests and catching bugs. Leaving all testing responsibilities to QA means feedback cycles will be long, prone to human error, and slower at scale. The reality is testing is a shared responsibility between engineers and QA. When you can efficiently test at the PR level, when a test fails, the developer knows immediately that their change created a bug. QA should not have to triage for hours and then assign a back bug to the developer that introduced it. The best time to catch a bug and create or update a test is during development, not after.

Consequently, some testing needs to be shifted left in the development cycle, which naturally becomes the engineers’ responsibility. By sharing the testing responsibility, engineers can drastically improve the quality of the builds that are sent to QA, who can then test edge cases and various user states.

Testing needs to be efficient

In a recent report, the number one thing engineers, both ICs and leadership, wish they could spend less time on was testing. This problem exists for three major reasons: 

1. Creating and maintaining tests is time consuming
2. Running the whole suite every time slows down development
3. Test creation and maintenance is faster as you build, not after

Waldo was originally built to solve the pain around how time consuming test creation is. However, to implement an efficient testing process, technology is not enough. Development teams need to start implementing a testing strategy by breaking test suites down into different components. This way critical flows can be tested frequently and other tests can be run nightly or before releases. With this strategy, teams can make testing scalable and efficient while also ensuring test coverage across their application.

Testing needs be transparent

We believe that, like other parts of the development process, testing needs to be more visible throughout the team. Tools like Github, GitLab, BitBucket have displayed the power of bringing more transparency to product development. When testing is done earlier, there is more clarity over what code changes led to a failed test. By making it easy to share test results, teams are better able to work together to find and solve issues in their code early and often. This will give them more confidence as they ship new features and products.

Conclusion

The future of mobile testing is continuous. Today’s announcement is a major step towards that future, one where mobile development teams do not have to choose speed or quality. We still have a lot more work to do but the Waldo team, in partnership with our customers, are working tirelessly to continue delivering on this vision. If you want to learn more, please reach out to us here.

We are hiring in all departments. Waldo has a product-led culture so our approach to work is both scientific and creative. We believe in experimentation, iteration, and objective analysis while also valuing ideas that are daring and unconventional. Egos are left at the door. It's never a competition for who is right but rather what is most right! We nurture the value of how to think, not what to think. You can see our open positions here or reach out to us at jobs@waldo.com.