The 8 Best iOS Testing Frameworks in 2023 For Mobile Devs

Mobile testing types
OverviewUnit Testing Explained & SimplifiedUnit Test AutomationFunctional Test AutomationBlack Box TestingIntegration TestingSmoke TestingAPI TestingAutomated Performance TestingMobile Usability TestingAutomated Security TestingMobile Regression TestingiOS Unit TestingAndroid Unit Testing
Mobile testing frameworks
OverviewData Driven
Modular
iOS
Android

Testing an app is an important part of any development process. In the highly competitive world of iOS apps, it’s essential to give users a great app experience. This includes a bug free experience while using the app. Earlier, most of the testing was done manually on physical devices.

But with the variety of iOS devices available today, it is difficult to test on all types of physical devices. An iOS app can run on an iPad, Apple watch, and Apple TV, in addition to an iPhone. All of these devices also come in different sizes and with different processors. With Apple charging a premium price for devices, it’s not feasible for a small startup to buy each of the devices for manual testing.

The solution to this problem is the iOS testing frameworks, which are easy to use. These testing frameworks are all you need to test on various simulated devices, which saves a lot of time and money.

In this post, you’ll first learn about iOS frameworks in general. After that you’ll learn about eight different iOS frameworks.

What are iOS Testing Frameworks?

Basically, you use a testing framework to create and run test cases. It enables you to eliminate manual testing, which saves both time and resources. Testers also conduct manual testing, use the app to check its functionality. The iOS testing frameworks have Apple specific functions like testing Apple APIs. iOS testing frameworks automatically generate detailed automatic reports, which is not possible in manual testing.

How to Pick the Right iOS Testing Framework for You

Before you look at the list of iOS testing frameworks, let’s pause a bit and teach you how to pick the best framework for your team.

Here are the main criteria you should use for this decision:

  • Cost
  • Supported platforms
  • Learning curve
  • Coding skill requirements

Cost

Price is a crucial factor you must pay attention to. The key step here is research. Compare the prices of different solutions—and pay attention to their features, unless you want to compare apples to oranges. And, of course, don’t forget the budget you have.

Negotiation also often pays dividends, so don’t be afraid to ask for a discount or some special condition.

‍

If you’re thinking about picking an open-source solution, always consider the TCO (total cost of ownership). Even with free tools, there are hidden costs you mustn’t overlook:

  • Steep learning curves
  • Infrastructure costs
  • Costs related to compliance and data privacy
  • Potential time spent acclimating to the open-source tool itself

We’re not saying you shouldn’t use open-source solutions. Instead, just take the concerns above into consideration when thinking about costs.

Supported Platforms

iOS is the indisputable market-share leader when it comes to the US and other regional markets. But globally, Android is king. There’s a high chance your organization also makes Android versions of their apps, especially if you target international markets.

If that’s the case, picking a testing framework that also supports Android brings some benefits. First, developers and/or QA personnel will have to learn a single tool, reducing the learning curve and making for a faster and more consistent testing experience.

Choosing a tool that supports both platforms also potentially reduces cost, since you don’t have to pay for two completely different frameworks.

Learning Curve

Learning curve is an essential aspect of making this choice. Before picking your iOS testing framework, try to assess its learning curve:

  • Read opinions and reviews from other users
  • Watch or read tutorials
  • Evaluate the framework for a while (in case it’s free or offers a free trial)

Reducing the learning curve as much as possible will reduce the time it takes for the professionals in your organization to be productive with the tool. That way, they can start creating tests and generating value sooner.

Also, time not spent learning the tool can be spent in activities potentially more valuable, thus eliminating opportunity cost.

Coding Skill Requirements

Some frameworks  are entirely code-based. Others are completely codeless. A third group mixes the two approaches. Which one should you choose? 

This is a decision you must make based solely on the makeup of your team. Consider how many professionals you have who are going to be involved in testing and assess their skills and aptitudes.

If you pick a testing framework that is code-based, that means developers will have to create the test cases or you’ll have to train your QA professionals (in case they don’t already know how to write code).

If that’s not a problem for you, a code-based tool won’t harm you. Otherwise, you might consider picking a codeless or hybrid solution.

Eight iOS Testing Frameworks

Now, let’s look into the 2022’s eight most popular iOS testing frameworks for developers.

Appium

Appium, created by Dan Cuellar using C#, is the most popular testing framework. It’s popular because developers can use it to test both iOS and Android apps. Developers also like that it’s open source and completely free to use. It was originally released in 2011 and converted to open-source in 2012.

Appium uses Selenium WebDriver APIs, which allows you to write test cases in any language of your choice. For example, you can write your application tests in Java, Python, JavaScript, or PHP, through client libraries available for each language.

As the most popular test framework, Appium has an active community for support. It also has a large repository of queries and answers about known issues in Stack Overflow.

Below is a sample of code in Appium.


text_fd = driver.find_element_by_id('Field1')
assert text_fd.get_attribute('value') is None
text_fd.send_keys('Hello Appium!’)
assert 'Hello Appium!' == text_fd.get_attribute('value')

TestProject

TestProject is a testing solution in which you have to record your test steps and then it generates test cases and results automatically. It’s one of the few test frameworks that automatically generates test and doesn't require you to write your test cases in a programming language, so you don't need to learn a programming language.

TestProject supports both iOS and Android platforms. You can use it with both iOS simulators and real iOS devices, like iPhone and iPad. But you need an Apple developer account to use TestProject on physical devices.

The feature of TestProject that developers like most is that they can test iOS apps on a Windows operating system. It doesn't require a Mac operating system or Xcode, which is a requirement for many iOS testing and development frameworks.

EarlGrey

EarlGrey is an iOS testing platform developed by Google. In fact, Google uses it to test its own iOS apps, like YouTube, Gmail, and Google Calendar. You can easily integrate EarlGrey with the Apple iOS testing framework, XCTest.

You can run it from the macOS IDE, Xcode, or from the macOS Terminal application. EarlGrey is completely open-source, but as of now, it has some 187 open bugs.

EarlGrey sends all its test data to Google Analytics, and you cannot easily turn this workflow off.

Below is a sample of code in EarlGrey.


var theMatch = true
  let matches: MatchesBlock = { (element: AnyObject!) -> Bool in
    if theMatch {
      theMatch = false
      return true
    }
    return false
  }

XCTest

XCTest is the official iOS testing platform. It comes built into the macOS IDE, Xcode. You can use it to perform unit testing, performance testing, and even UI testing.

XCTest is geared towards iOS app developers and you write test cases in Objective-C or Swift. Testers who only know languages like Python or Java and use Selenium testing may find using Objective-C or Swift problematic.

Below is a sample of code in XCTest.


class StringTests: XCTestCase
{
    func testShouldPass()
    {
        let num1 = 220
        let num2 = 100
        XCTAssertEqual("\(num1) plus \(num2) is equal to \(num1 + num2)", "220 plus 100 is equal to 320", 
"String Interpolation Test")
    }
}

Detox

Detox is another completely open-source testing tool for the iOS platform. You need to write test cases in JavaScript, which one of the most popular programming languages on the planet.

Detox testing uses an iOS simulator; and it has no support to run tests on physical devices, which can be a drawback for some users.

Below is a sample of code in Detox.


describe(‘Registration flow', () => {  
  it('should register successfully', async () => {
    await element(by.id('email')).typeText('john@example.com');
    await element(by.id(‘user’)).typeText(‘jo’hn);
    await element(by.id('password')).typeText(‘welcome’);
    await element(by.text(‘Register’)).tap();      
    await expect(element(by.text(‘Registered successfully'))).toBeVisible();
  });  
});

Calabash

Calabash is an iOS testing framework that takes a different approach from the others. You write Calabash tests in Cucumber, making test cases very easy to write, especially for non-tech people, who find writing test cases in Cucumber easier than in other programming languages. Cucumber is a programming language in which you can write plain English test cases. So, with Calabash, you don't have to learn any programming language and can write generic test cases (see code below).

Calabash is very stable and supports iOS simulators and physical devices.

Below is a sample of code in Calabash.

Feature: Registration feature

  Scenario: As a new user i can register in the app

    When I press “Register”

    And I enter my username 

    And I enter my email

    And I enter my password

    Then I see “You are registered successfully”

OCMock

OCMock is not a complete testing solution, but instead, you use it to create mocks. Mocks, special code in which you can simulate things, like API calls for example, are very popular in the testing community. Instead of using a real API call, which costs money, you simulate an API call. Because Xcode has no built-in support for mocks, we use OCMock to create mocks.

OCMock stands for Objective-C Mock. And as the name suggests, you write your mocks in Objective-C. It is completely open-source and has very good documentation.

Below is a sample of code in OCMock.


id loginDefaultsMock = OCMClassMock([NSUserDefaults class]);
OCMStub([loginDefaultsMock stringForKey:@“AppURLKey"]).andReturn(@"http://url.app”);
OCMStub([loginDefaultsMock stringForKey:[OCMArg any]]).andReturn(@"http://url.app”);

KIF

KIF stands for "Keep it Functional." KIF uses the Apple iOS testing framework, XCTest, to create automated tests in the XCTest format. You write your test cases in Objective-C, which along with Swift, is the language for iOS app development. Developers nowadays don't favor Objective-C very much, which limits use of KIF. They prefer to use the more modern Swift programming language.

Although KIF tests run faster than tests written in Swift using XCTest, the language restriction of using only Objective-C is an issue.

Below is a sample of code in KIF.


- (void)testSuccessfulRegistration
{
    [tester enterText:@“name@example.com" intoViewWithAccessibilityLabel:@“Register Email”];
    [tester enterText:@“name” intoViewWithAccessibilityLabel:@"Register user” name];
    [tester enterText:@“welcome” intoViewWithAccessibilityLabel:@"Register Password"];
    [tester tapViewWithAccessibilityLabel:@“Register”];

    // Verify that the registration succeeded
    [tester waitForTappableViewWithAccessibilityLabel:@“Registration done”];
}

Conclusion

In this post, you learned what iOS testing frameworks are, and then you learned about the eight best iOS testing frameworks. You learned about the benefits and drawbacks for each of them, and we provided sample code for each.

We can also use the scriptless testing platform Waldo instead of writing all test cases manually. We just need to upload the APK or IPA file, and then Waldo can automatically generate and run your testing. Create a free Waldo account here to test its features.‍

This post was written by Nabendu Biswas. Nabendu has been working in the software industry for the past 15 years, starting as a C++ developer, then moving on to databases. For the past six years he’s been working as a web-developer working in the JavaScript ecosystem, and developing web-apps in ReactJS, NodeJS, GraphQL. He loves to blog about what he learns and what he’s up to.