User State Management in Waldo

In Part 1 of this series, we introduced you to Waldo on Waldo (WoW), our internal process for testing our web app using our own solution. Now, in Part 2, we'll delve into the crucial aspect of user state management and how we tackled it within WoW.

Fresh User Accounts vs. Pre-existing User Accounts

When it comes to testing, we have two main approaches at our disposal: starting with a fresh user account for each test or reusing pre-existing user accounts. Let's take a closer look at the pros and cons of each strategy.

Fresh User Accounts

Using a fresh user account for each test has its benefits. One major advantage is that it eliminates issues related to concurrency. By starting with a clean slate, we can avoid conflicts that may arise when multiple tests attempt to update a user's account state simultaneously. This ensures accurate and reliable test results.

However, there are considerations to keep in mind. Creating a new user account for every test means we have to recreate the user's state from scratch. This can be time-consuming, especially if the user's state involves complex settings, preferences, or data. In Part 3 of this series, we will explore how we addressed this challenge and optimized the process of recreating user states.

Pre-existing User Accounts

Alternatively, we can choose to reuse pre-existing user accounts for our tests. This approach can save time since we don't need to create new accounts or set up user states for each test iteration. It allows us to focus solely on the specific functionality we want to test without the overhead of user creation.

However, there are some considerations when using pre-existing user accounts. One potential drawback is the risk of side effects. If multiple tests share the same user account and modify its state, changes made by one test could impact the results of subsequent tests. Therefore, careful planning and isolation of tests are crucial to avoid interference and maintain reliable test outcomes.

In summary, we initially chose to use fresh user accounts to ensure reliable results. However, as we progressed, we found a solution to speed up test setups by leveraging data cloning, which we will explore in Part 3 of this series. This enhancement allows us to optimize the testing process while maintaining the accuracy and integrity of our test results.

Bypassing Plan Upgrades

Let’s go through the example of bypassing an upgrade sales cycle in order to test certain account levels for WoW.

In Waldo, we encountered an obstacle: accounts need to go through a sales cycle to upgrade to a new package, and this is not a self serve motion. To efficiently test with WoW, we needed a way to bypass this step while accounting for the upgrade.

There were a few options: we could pull from an existing user with that account level, or we could generate a new user that would have that account level automatically to ensure a fresh state.

We created a mechanism to generate a fresh user while bypassing the account upgrade process by creating email patterns that would generate the user in a specific plan tier.

Yes, we understand it may raise security concerns, but rest assured, this mechanism is not activated in our production environment. This only works on the staging environment WoW is testing on, and access to that environment is limited to our team only.

This allows us to perform thorough testing without being hindered by subscription requirements. It's like having a secret backstage pass to the llama show. We created this bypass through email patterns because that's how we uniquely identify our users. The same technique could also apply to phone numbers or other user attributes.

In Part 3 of this series, we will explore how we enabled data cloning to further expedite the creation and execution of tests within WoW.

Stay tuned for the next installment as we continue our journey through the innovative testing process of Waldo on Waldo.