Calabash vs. Appium: How Do They Compare?

In this post, we're going to learn about the mobile testing frameworks Calabash and Appium. There's quite a bit of difference between both of them. Calabash is used mainly for acceptance testing, whereas Appium is used for automation testing. 

We'll briefly discuss the two frameworks and then go over their advantages and disadvantages. Then we'll see some usage examples for both of them. Finally, we'll give our verdict on the best framework. 

What Is Calabash?

Calabash is a completely free-to-use, open-source mobile testing framework. It's used to test both iOS and Android mobile apps. You can also contribute to its development, and it has a good community behind its support and maintenance.

It's an acceptance testing framework, meaning it’s mainly used for functional testing in agile development. Developers can to test their small tasks, also known as stories, with it. 

We write Calabash tests in Ruby using Cucumber, which uses a BDD (behavior-driven development) approach. The test cases are regular English statements, which are written following a language called Gherkin. We also need a bit of Ruby programming to execute them. 

Advantages of Calabash

• We can have a common feature file for both iOS and Android testing. The feature file is a file in which the code is written in the Gerkin language. After that, the step definition file, which is written in Ruby can be different from both.
• Calabash is part of BDD, in which the behavior of the app is decided first. So, the QA team and stakeholders can write the test cases of different components in English statements. They don't need to know any complicated languages like Java or Python.
• Calabash tests can be run on iOS or Android devices. They can also be run on simulators. Besides this, they can run concurrently on several mobile devices.
• There's very good community support in Calabash, which is a must for any popular open-source project.
• The DevOps integration of CI/CD (continuous integration/continuous development) can be done easily with Calabash. This means the moment the test cases are passed, the code can be pushed into production.

Disadvantages of Calabash

• Setting up Calabash on a system is complicated. We need to have Ruby and Android Studio properly installed on the system. And we also don't have tools like Appium Doctor to check if installation is done properly.
• Calabash features files are written in English statements by QA engineers, but the step definition file requires a fair bit of Ruby knowledge and is generally written by developers.
• It cannot be used for automation testing, which is used to automatically test that all features are working fine.
• Only the Ruby language is supported. No other major languages, such as Java or Python, are supported.
• There's no support for mobile-based web apps, which is not the case with Appium. Only mobile apps can be tested with Calabash.

What Is Appium?

Appium is an open-source and free-to-use mobile app testing framework. It can be used to test any type of mobile application. Appium can test apps written using the native languages of Kotlin, Java, Swift, or Objective-C. It can also test mobile apps created using development kits and frameworks like React Native or Flutter. 

Test cases can be written in a variety of languages, such as Java, Python, JavaScript, or PHP. In an earlier post, we wrote test cases in JavaScript. When we run the test cases, Appium will show us the result. 

Besides this, the app will run automatically on the connected device and show the user interactions written in the test case. It'll seem like a real user is participating in the interactions. This is quite similar to Selenium automation testing, which is used to test web apps. One thing to know is that Appium is based on Selenium. 

Advantages of Appium

• Appium tests are generally written by QA engineers in a language of their choice, like Java or Python. Once they write these test cases in a script, it can be used to test both the iOS and Android apps. These apps could have been written in different languages like Swift (iOS) or Kotlin (Android).
• It provides the flexibility to write test cases in the language of your choice like Java, Python, JavaScript, PHP, and many more.
• QA engineers familiar with writing test cases in Selenium can easily write test cases in Appium. The format is quite familiar, because Appium is based on Selenium.
• There are other open-source tools to help work with Appium. Appium Doctor helps with the installation of Appium, and Appium Inspector helps with finding element IDs with ease for testing.
• There's very good community support in Appium. The open-source community regularly adds features to the Appium Server, Appium Doctor, and Appium Inspector.
• It also has paid tools like Appium Studio, created from the open-source tool. This tool is the easiest to use, and it has a lot of additional features. Also, a community version is available for free.

Disadvantages of Appium

• Appium requires knowledge of a complex programming language like Java or Python. Most of the time, the QA team doesn't know these languages. So, it's difficult for them to write test cases in Appium, unlike Calabash, where test cases are written in English statements.
• The setup for Appium on Windows and Mac is different and complex. It requires Java to be installed properly. Besides this, Android Studio (Windows) or XCode (Mac) should be installed and configured. That is the reason we have tools like Appium Doctor, which checks if installation is done properly.
• Setting up CI/CD pipeline with Appium is not straightforward. The DevOps team might use different languages (like Perl) than the QA team (who might use Python), and this can cause complexities in setup.
• It doesn't support testing mobile devices that run older operating systems. Android phones with Android 4.1 and lower are not supported, and iPhones with iOS 12.2 or lower are not supported.

Similarities and Differences

Appium and Calabash are both used for mobile app testing. There are other tools for web app testing, but these are the tools specifically created to test mobile apps. Both are open source, with quite a vibrant and active community. Because of this, new features are added into both regularly and bugs are fixed quickly. And both of them can test all types of mobile apps written in a variety of languages. 

The main differences are that Calabash does acceptance testing, whereas Appium does automation testing. Calabash test cases are written in English statements in a language called Gherkin, whereas Appium test cases can be written in a variety of languages like Java, JavaScript, and Python. 

Testing Apps with Calabash

Apps can be tested with Calabash by writing test cases in Gherkin. In our earlier post on testing android apps with Calabash, we first did the setup on a Windows machine. The setup includes installing Ruby and Calabash on the Windows machine. After that, we wrote the test cases in English in the feature file to test a simple login feature. Finally, wrote a few lines of Ruby code in the step definition file to test the app. 

Testing Apps with Appium

Since Appium test cases can be written in a variety of programming languages, we have many ways to write our tests. In our earlier post on Appium for iOS, we learned to do an Appium installation on Mac, and we followed it by writing automated test cases in Java. 

In another post on testing Appium with Python, we learned to do an Appium installation on Windows. And followed it by writing automated test cases in Python. Then, in yet another post on testing React Native apps with Appium, we learned to do an Appium installation on Windows, and we followed it by writing automated test cases in JavaScript. 

Verdict for Best Framework

Although both Appium and Calabash are used for different types of mobile app testing, Appium is the clear winner here. It has far more features than Calabash, including more supporting tools. Besides this, it also has a paid version called Appium Studio, which has far superior support and features. 

What You've Learned

In this post, you learned about the mobile testing frameworks Calabash and Appium. You learned about Calabash first, followed by its advantages and disadvantages. After that you learned about Appium, followed by its advantages and disadvantages. You also saw various app testing examples with both Calabash and Appium. Finally, you learned that Appium is the winner when comparing the two tools. 

But the setup for both is complicated. Besides this, writing test cases in Appium is also complex, and knowledge of a programming language is required. To test your app without complicated setups and test cases, you should try Waldo. This testing platform requires you to provide the APK or IPA (iPhone Application) file. After that, you can interact with the mobile app like a real user. Waldo automatically generates test cases and shows results in a nice graphical format.