How quickly can you do a check to see if your application still performs its most basic functions? If you just mentally went through the steps necessary to do this, you designed a smoke test. 

Smoke testing in software engineering is a testing method that verifies an application performs a minimal set of functions without which further testing or release to production is impossible. It's usually done prior to releasing or before continuing with subsequent, more detailed tests. Testers use it to check the critical functionalities and determine the stability of the software product and to decide if they should execute their more elaborate tests of edge cases and less important functionalities. 

Smoke Testing vs. Regression Testing vs. Sanity Testing

There are many types of testing, often with only subtle differences between them. When speaking of smoke testing, we might find it confusing to understand how it relates to regression testing and sanity testing. Regression testing ensures that a system continues to work as it did previously, even after a modification. Regression testing involves running previously designed tests that we know have been running successfully in the past.

If a test fails, some new change must have broken it (i.e., the developers introduced a bug). Smoke testing (sometimes called “build verification testing”) is a subset of regression testing. Smoke tests are a small collection of functional tests that verify the most important functionality of the application. Smoke tests should run very fast so that testers know quickly if they should continue testing. Smoke testing is sometimes used interchangeably with the term "sanity testing." Both are meant to determine if further testing is sensible or even possible. However, sanity testing can also refer to a quick check of a specific change. Smoke testing is then a fast way of making sure the new code didn't break anything else. 

Why Is Smoke Testing Important?

We'll get into the advantages of smoke testing below. But we can already mention that smoke testing helps testers in their daily workflow. A suite of smoke tests will provide fast feedback on whether or not they can start their work. If a smoke test fails, they can decide not to test all or part of the application.

In these fast-paced days of modern and agile software development, teams want to reduce waste. Wasted time is a frustrating experience, but it also comes with financial cost and opportunity cost. It costs money, and the tester could have spent their time more productively, adding value somewhere else. 

Advantages of Smoke Testing

Let's dive deeper into the advantages of smoke testing. 

Smoke Testing Is Fast and Effective

The idea behind smoke testing is that the test suite is fast, and that it focuses on the major functionalities of the application. As such, smoke testing provides quick feedback about the basic functionality or the most important bits of the system.

Smoke Testing Is Flexible

Smoke tests can easily evolve over time. Because the test suite should remain minimal, the team can add, remove, or modify smoke tests easily. If the expected outcome of a functionality changes, there won't be hundreds of tests to update. If the team wants to add a new functionality to the smoke tests, it shouldn't be too much work. 

Smoke Testing Can Be Automated

Like other regression tests, the team can (and should) look at automating the smoke tests. Usually, smoke tests will take the form of API tests or UI tests. Both of these can be automated, providing the testers with an even quicker way of gathering feedback about the state of the system. Automation also rules out human error during testing, and it allows the team to incorporate the smoke testing suite into a build server that runs regularly. This way, every new build can quickly provide feedback.

Smoke Testing Saves Time in QA

Quality Assurance (QA) can be a lengthy process. A large application might require hundreds or even thousands of tests to fully verify that it's in working order. Even if the team automates these tests, the amount of tests will cause a full run to take some time. And even with automated tests, it's still beneficial to perform manual tests. Smoke testing can stop testers from starting the larger test suite, saving their precious time. 

Smoke Testing Improves Your End Product

Like all testing, your end product will benefit from smoke testing. The quick feedback allows teams to make progress faster, but testing the major functionalities of your system ensures that at least those important bits continue to work for the end user. 

Types of Smoke Testing

We can break smoke testing down into three major types, which we'll cover in more detail: manual smoke testing, automated smoke testing, and a hybrid form. 

Manual

Testers can perform smoke tests manually. This involves interacting with the application by using the user interface or by using tools to call the API. Obviously, this is a time-consuming process. But if the application is small enough, this can be an acceptable process. 

Automated

Once an application reaches a certain size (and this is quite early on), it pays to automate the smoke tests. A test automation tool can then run through the necessary steps to perform the smoke tests and verify the outcome. Teams can easily automate calls to an API with one of the many tools that exist today. The UI is slightly harder to test automatically, but even for user interfaces, you can still find tools that make it easier (like Waldo). 

Hybrid

Obviously, teams can use both manual and automated smoke tests. Maybe certain tests are difficult to automate. Or maybe the team wants to keep a human touch to the testing. Another reason could be that certain smoke tests start out as manual tests and are automated later on if deemed appropriate. 

How to Perform Smoke Testing

The first step to smoke testing is part of overall test case management, deciding what to test. We don't just want to test all features and all user scenarios. As stated earlier, smoke tests are a form of regression tests, but smoke test cases should only test the key features of the application. So the key is to decide what the major features are.

Smoke tests should be limited in number so that they provide near-immediate feedback. After the team decides what to test, they should design the tests by determining the different steps to take and pieces to verify. If the team chose to automate the smoke tests, they have to implement the tests. How they do this depends on the chosen tool. Then testers can run the tests. Either they run through the steps manually, or they can trigger the tests with a simple command. Finally, they should analyze the test results and pass any errors on to the people who can fix them. Once the smoke tests pass, testers can move on to the next part of their task: testing existing and new features, including all edge cases and minor features. 

Things to Remember When Smoke Testing

Smoke tests really bring value to the table if the team runs them against a live version of the application. Ideally, this is in a QA environment. Smoke tests should test the entire application, but they should be performed before deploying to production. That is why smoke tests work best in a QA environment, although you could also run them against your production environment to be sure. 

Because smoke tests test a minimal set of features (the most important ones), there is no need to mock any components. Special test scenarios (e.g., testing when external services are down) are not a good fit for smoke testing. As previously mentioned, teams should automate their smoke tests. This will make them run the tests more often. It also allows them to incorporate the smoke tests into the CI/CD build. 

Because of automation, the entire test of automated tests can be run very fast these days, possibly removing the need for smoke tests altogether. But for certain teams and organizations, they still provide value. For example, when the entire suite would take hours, it can be useful to get fast feedback from the build server if anything important broke. 

Smoke Testing in Review

Smoke testing is a useful addition to any team's test suite. It saves time by giving them fast feedback about the state of their system. If smoke tests fail, they shouldn't continue with any further testing. Rather, they should look into the failing smoke test first, because it means an important feature is not working as expected. We can run our smoke tests manually, but we can also choose to automate them. 

Automation is important in a DevOps culture, because it allows us to run the test suite quickly, easily, and continuously. This way, we're alerted early on when something is wrong, which saves even more of our team's precious time and avoids issues going into production. Even then, manual smoke tests can still be valuable. Smoke tests verify the most important features of an application. They can test the API or the user interface. Either way, smoke tests should include as many components of the application as possible.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.