Your application can perfectly implement all the required use cases, but if it doesn't perform well under stress or if it doesn't respond fast enough to user input, users won't be happy with it. Performance tests can help developers identify when a change is introduced that slows down the application, or which parts of the application suffer from bad performance. 

Performance testing is a common software testing practice among well-performing teams. Without it, they couldn't make their applications speedy, scalable, and reliable. Performance tests are necessary to ensure that an application can handle thousands of users simultaneously and still give them a good experience. But before starting with performance testing, it's good to know what kind of performance tests there are, what benefits they provide, and how to get started. 

Types of Performance Testing

There are several different types of performance tests. Each type addresses a different scenario. In each case, the test can help us identify if there is a performance problem with our application, and where we should investigate. 

Load Testing

Load testing is a type of performance test that mimics real world conditions. A load testing tool will hit the application with a realistic number of requests or inputs during a short amount of time (typically several minutes). This realistic amount doesn't have to be an average amount. It can also represent the highest peak that the application has seen in the past. 

Stress Testing

The terms stress testing and load testing are often used interchangeably. But while a test case in a load test tries to mimic realistic conditions, stress testing will increase the load until the application is using its maximum resources. Often, testers will increase the load in steps until the application breaks. This tip-off point is then the maximum load the application can handle. Of course, they can define other thresholds in their test scenarios that the application shouldn’t cross (a maximum latency for example).

Spike Testing

Spike testing is similar to stress testing. But rather than gradually pushing the system to its limits, the test mimics regular app usage and then simulates short peaks of requests. The goal is to see how the system responds to a sudden increase in traffic, and to see how well the system recovers after the peak. Spike testing is important for applications that expect a spike in usage at certain points in time. A well-known example is the traffic a ticket sales website might receive. They often experience large spikes at the moment when customers can start buying tickets for popular events. 

Scalability Testing

The previous three types of tests are closely related. In those tests, we test the system under a certain amount of load. The idea is to determine how our application handles the given load. We could perform these tests for several reasons. Maybe users have been complaining about slow performance, and we need to identify the problematic areas of our code. Or maybe we want to know how many new users our application can serve before the experience starts to deteriorate. This next scenario is a case of scalability testing. In scalability testing, we test to see how much our application scales. When we start writing our software, we can't predict how many users will be using the application in 5 years' time, for example. Scalability testing gives us an insight into how much we can stretch our application before larger architectural changes are necessary. 

Endurance Testing

Endurance testing is similar to spike testing, except that in endurance testing, the test will hold the spike for a longer period of time. The idea is to see how the system reacts to a high load that extends for a longer period. Some applications may be able to handle high spikes of requests well, but their performance may deteriorate over time if the spike continues to last. For example, a memory leak often only manifests itself after several hours or even days of usage. 

‍

Volume Testing

Another interesting type of performance testing is volume testing. In volume testing, we increase the volume of one of our application's resources. For example, we might increase the amount of data in the database and measure how our application handles this. Or we could feed the system a very large file when it normally works with smaller files. If the system doesn't handle this well, and we expect these larger volumes to be realistic in the future, we should investigate and fix the issue. In our examples, this could mean tweaking database queries or changing the way we read the file. 

Benefits of Automating Performance Testing

Now that we've covered the different types of performance testing, we can start to see what advantages performance testing gives us. Companies run performance tests to ensure that the user experience of their application is good enough. The specific reasons depend on the application and its usage, but think of things like applications with extremely complex calculations, high spikes in usage, or large databases. Companies want to be sure that their application is and remains pleasant to use, and that it solves the user's problem as swiftly as possible.  If there are any performance issues, the team can then put effort into optimization.

While performance tests can be done manually, it pays to automate them, like many things in a DevOps culture.It makes the team more agile as it can get fast feedback about the performance and run the test scripts regularly.  Let's cover some of these benefits in more detail. 

Prevent Performance Relapse

Teams can often focus on new features so much that they forget to look out for warning signs that the application performance is degrading slowly. As time goes on, they can reach a sudden tipping point where the performance drops suddenly and extremely. Unfortunately, many teams then have to go into a firefighting mode to quickly find and solve the issue. Needless to say, end users aren't too happy when that happens.

This can be avoided by running regular performance tests. These tests will tell the team early that something is off and must be investigated. The tests will notify them before end users really notice. 

Because this effort can only be successful when the tests are run on a regular basis, it's in the team's best interest to automate these performance tests. This way, they can run quickly and easily. They can integrate them into their daily build and create dashboards that make the current performance level visible. These things are all practically impossible when the performance tests are run manually. 

Improve User Experience

Automated performance tests ensure that performance is a first-class citizen in the team's software development process. In turn, this ensures that the quality of the user experience remains high and even increases. By analyzing the results of the performance tests, the team can identify areas where they can improve the performance, and consequently, the user experience. 

Avoid Launch Failures

Too often, new versions of an application have been launched only to see the performance drop drastically, and end users complain loudly. Automation makes it easy to run these tests and postpone a potential bad release. The team can then fix the performance issues and release with confidence. Executing performance tests manually increases the likelihood of not running them before a release. 

Eliminate Bottlenecks

Performance testing can help identify problematic areas of the code. In a REST API, for example, it can show a team which endpoints should be investigated further. A slow user experience in a mobile app could be caused by just a few HTTP endpoints. Identifying the endpoints that are slow to respond is the first step toward fixing the problem. The engineers can then investigate further to find the exact cause of the problem and fix it. 

Avoid Manual Missteps

Manual performance testing is definitely possible. But like all types of testing, automation has the benefit of avoiding human mistakes. As humans, we're prone to forgetting certain steps or executing them wrongly. When we automate the process correctly, the test system will perform the test the same way every time. 

Shift Performance Testing Left

The idea of "shifting left" is to perform tests as early as possible in the software development lifecycle. It's an idea that is popular in DevOps. Picture the software development lifecycle as a flow from left to right. Each feature goes through several stages, from gathering the requirements to deployment to production. 

If we're only notified about performance issues in production, we're effectively using the production environment as our performance testing environment (and even letting our end users do the performance testing). The idea behind "shift left" is to move quality checks as left as possible. Unit tests, modern analysis techniques like event storming, and automated security scans are examples of shifting left. Automated performance tests are also a way of shifting left. Instead of waiting for performance issues to arise in production, we can run our performance tests against the QA environment, or even run them on the developer's computer. 

How to Do Performance Testing

Before a tester starts performance testing, they need to know what metrics they want to analyze. This depends on the application. For web applications, testers will typically look at response times of key endpoints. For other application types, CPU or memory usage may be relevant. And as we saw, the way the users are expected to use the application determines the performance test type. 

After this, the tester must decide on the acceptable values for the metrics. It's also possible that the tests are more exploratory, and that the acceptable values are determined by the outcome of the first test run. For example, the team might decide that they want to increase response times by 20% after initial testing. 

Next, the team has to set up the test environment, install the necessary testing frameworks (like JMeter or Loadrunner), and then create the tests. They must also decide on how regularly they want to run the tests, and what the trigger must be. They could choose to run the tests daily, or shortly before a release, for example. 

As a last step, the team must act on the test results. If the performance is degrading, they'll have to make plans on how to improve this. When the team can repeat this and iterate on it, they'll reap the benefits that we covered earlier. 

What to Look for in a Performance Testing Tool

There are many performance testing tools available. Some are general and offer a wide range of possibilities, while others are opinionated. Here's what to look out for. 

• Investigate whether the tool serves your needs. For example, a memory profiler can't analyze web response times. Depending on what part of the application you want to test, and which performance test type you want to run, you'll be able to trim down your available options.
• Check whether the tool can be integrated into your software development process. Ideally, you want your performance tests to be automated, as we covered previously. If you need to log in to a website and click several links before you can run your tests, you won't be able to integrate it into your continuous integration flow.
• Pricing, licensing, and support are very important to many companies. It's better to know your needs for these requirements before you start investing time into the tool. And you'll want to get approval from management, of course.

Conclusion

Performance testing is an important part of the test process for modern applications where scalability and user experience are important. Automated performance testing is a logical next step after running some tests manually. 

Automated performance tests can be run more often and can be incorporated into a team's daily workflow. This will give them the many benefits that these types of tests provide. But in the case of mobile applications, things can get tricky. Performance tests must be run on different devices, platforms, resolutions, and network speeds.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.