Functional Test Automation

In software development, functional testing is an important step before we release our software to end users. Without it, we have no way of knowing if we've broken anything. Functional testing won't catch all bugs, but not doing functional testing is just unprofessional software development.

It's a first step toward the shift-left mantra of DevOps.  We don't want our end users to be our testers. Rather, we want to catch issues as early as possible. One simple step toward that goal is to at least test our software before we ship it. In many companies, this means having one or more testers run through a test scenario.

Obviously, this doesn't scale well. Once the application starts having a reasonable amount of features, it takes more and more time to test everything. Functional test automation is the answer. Automation is key to DevOps, and automating application testing will help teams run more tests in less time. This in turn gives them more time to develop new features while still maintaining a higher level of software quality.

But because software testing is a broad space, let's first look at what functional testing is exactly. Then we can look at the difference with nonfunctional testing, some challenges we face, and how we can automate functional testing.

Functional Testing Definition

Functional testing means testing the functionalities of your software from an end-user perspective. If the application has a user interface, the testing process consists of opening the application, using it, and verifying the outcome. If the application is only an API (e.g. a web service of a software library), functional testing means calling the public API and validating the results.

Functional testing is often done by dedicated testers, though in smaller teams, the developers may perform functional testing as well. The testing process is fairly simple. To get started, the tester will design a test scenario first: what will they be testing and how? Then they'll start the application and perform the required steps.

As they go over the steps, they verify the outcome. In case the application doesn't have a user interface, this validation consists of looking at the result returned by the software and any external effects (like data added to a database, for example). If the application does have a user interface, there's a little more to it. Functional testing can also bring up issues with the user experience. Maybe everything works as intended, but the experience just isn't pleasant or efficient.

Functional vs. Nonfunctional Testing

If there's such a thing as functional testing, there must be such a thing as the opposite: nonfunctional testing.

Functional testing includes the following test types:

• Unit testing
• Integration testing
• API testing
• End-to-end testing and UI testing (sometimes called front-end testing)
• Smoke testing

The above types of testing can all verify functional parts of the application without being concerned about how the functionality was implemented.

Nonfunctional test types include:

• Performance testing (e.g. response times, load tests, or memory profiling)
• Security testing
• Usability testing
• Recovery testing

The idea behind nonfunctional testing is to test aspects of the system that may have a detrimental effect on the user experience, but that don't necessarily mean the user can't perform their tasks with the software.

For now, let's focus on functional testing.

Importance and Uses of Functional Testing

Functional testing is important because it helps ensure users get the product they want. If bugs make it into production, they're usually more costly to fix than when we can catch them earlier.

Of course, this means our tests must be as thorough and complete as possible. The less comprehensive they are, the fewer bugs we will find, and the less confidence we will have in our code. Over time, we may even lose the confidence to add new features!

Functional testing is an integral part of any modern development cycle. A mature test suite will provide assurance that the software can be shipped to end users. New features should be accompanied by new functional tests. Older tests for existing features should still be executed to ensure that these features don't stop working.

This also underlines the importance of test maintenance and test management. Tests aren't written to be forgotten. They must evolve with the product. Sometimes they need to be removed or completely rewritten. Testers can also figure out new test scenarios for existing features.

A complete test suite allows teams to optimize the product before launching the new version.

Functional Testing Challenges

But functional testing isn't always easy. As mentioned above, we want to create a test suite that is as complete as possible. This alone is quite a challenge. But even if you have a complete set of test cases, it can be time consuming to execute them all manually.

Another challenge that functional testing faces is that some cases are difficult to test. You might want to test your software in case certain external services are offline. Or maybe you want to test your mobile app when it has an unstable internet connection.

This makes it difficult to find all possible bugs. You might only know about the bug when the end user files a frustrated bug report, if they even do at all!

In our world of fast-paced software development, your end users expect regular releases. This means you also need to test regularly. Manual testing won't scale very well. A manual testing process is too time consuming. It's also too error prone, since humans make errors. And if you require a cross-browser web application, testing it in the many different browsers manually is near impossible. Automating your functional tests solves this problem.

Automation reduces possible human errors, decreases testing time drastically, and frees up time for testers so that they can focus on designing tests instead of executing them. Automated tests also allows teams to adopt continuous testing: running tests continuously in the software delivery pipeline (e.g. after every change or before every release).

‍Functional Test Automation

The above is why functional testing is best automated. But there are cases where it still pays to perform manual testing:

• To evaluate the user experience (i.e. the look & feel).
• To explore the application before designing an automated test.

But mostly, you'll want to automate your tests. So how do we go about that?

Start by thinking about your process. Who will design the tests? Are they part of the specification handed to developers? Or will testers design them after the feature is finished? How often and where will the tests run?

Next, pick a test automation tool. This depends on the type of software you're building. If it's a web API, you could use a tool like Postman or SoapUI to make automated calls to your API. Functional tests against a web application can be tested with the open source Selenium web driver. And you can test your mobile apps with Appium or Waldo.

Depending on the tool you chose, you might want to think about your testing framework. Some tools have recording features that can produce linear tests: They run through all the steps that you took when you recorded the test. Others, like Waldo, have options to reuse steps across different test cases. This makes it significantly easier to build a complete test suite, because there are many common steps in an application (like logging in, for example).

After deciding on a tool, it's time to set up your testing environment. How to do this will depend on your specific scenario. Examples include:

• Starting the software on your machine.
• Starting the application on a test server.
• Installing the app on a collection of mobile devices.

You may also have to initialize test data to support your tests.

After that, you can start with the core of your test automation: writing test scripts. As stated before, you want a high level of test coverage. But it's fine to start small. If you're designing tests for a new application, your suite can grow along with the application. If you already have a mature application, you probably won't be able to make test scripts for the entire application at once. And that's fine.

Then it's time for the test execution. The chosen test automation tool can now run the tests and produce a test report.

Finally, you should take a look at the test report and perform any necessary actions. Either all tests pass, and you can release this version of your software, or some have failed, and you'll need to investigate exactly why they failed.

Functional Testing Solutions

Let's take a brief look at some functional testing solutions.

Selenium IDE is a popular one for web applications. With its record and playback functionality, you can record your actions in a browser and then have Selenium perform them again automatically in any other browser. This makes it easy to create tests for web applications and test your application across many browsers.

Appium is based on Selenium, but it's is aimed at mobile applications. It supports writing tests in many popular programming languages like Java, C#, PHP, and Python.

Waldo is another functional testing tool for mobile applications. Like Selenium, it supports record and playback. It features an easy UI so that non-technical profiles can also create tests, with reusable groups of steps and tolerance for flaky tests. Like many other tools, it integrates with your CI/CD pipeline, but it can also publish results to Slack or GitHub. It includes a dashboard, runs tests in parallel, and allows you to run your tests on multiple devices and screen sizes (without having to re-record your tests).

Automated Functional Testing in Review

Functional testing is basically testing your software in a way that mimics how the end user would use it. For any normal-sized application, this would take quite a lot of time if done manually. Especially for web and mobile apps that run on many devices, browsers, and screen sizes. That is why DevOps teams automate functional tests.

The team can design tests to verify that a feature has been implemented correctly. Ideally, they'll run these tests regularly, as part of a CI/CD build process, and definitely before a new release.

Many automated testing tools have significant advantages for development teams, such as:

• Record and playback functionality that allows non-technical people to create tests.
• CI integration that makes it easy to run tests on every code change, providing early feedback on any regression bugs.
• Reusable steps to create modular tests and make test maintenance easier.
• The ability to run tests on different platforms to ensure the app works for all users.

If you haven't automated your functional tests yet and are interested in taking a peek at Waldo, why not book a demo to see how Waldo can help.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.