Testing End-to-End from Bitrise with Test Reports and Waldo Scripting

In my previous two articles, I described step by step:

1. How I took an existing iOS app and added support for end-to-end testing with Waldo Scripting by using a GitHub Actions workflow. (See here)
2. How I took that same app and added support for testing end-to-end locally with Waldo scripting. (See here)

An intrepid reader of my articles wanted to know how to run an end-to-end test with Waldo Scripting in Bitrise and then export the results to the Bitrise Test Reports add-on. This presented a challenge that I was more than willing to accept.

What follows is a step by step guide to how I went from the simple test results as displayed in the Bitrise build log:

‍

to the test results as displayed in the Bitrise Test Reports add-on:

‍

Preliminaries

Continuing on from where I left off in the previous articles, there were only two preliminary tasks that I needed to address:

1. Make sure I had access to a Bitrise account.
2. Create a new project from the Bitrise dashboard that could access the GitHub repo for my app.

Once I took care of these preliminaries, I was ready to get down to business.

Writing the Bitrise Workflow

I found writing the Bitrise workflow to be relatively straightforward — I simply translated all the actions in my existing GitHub Actions workflow (see “Writing the GitHub Actions Workflow” in my first article) to the (more or less) equivalent Bitrise steps.

Here is what I ended up with:

format_version: "13"
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git
project_type: ios
app:
  envs:
  - BITRISE_PROJECT_PATH: TravelSpot.xcodeproj
    opts:
      is_expand: false
trigger_map:
- workflow: test-e2e
  push_branch: main
- workflow: test-e2e
  pull_request_source_branch: '*'
  pull_request_target_branch: main
workflows:
  test-e2e:
    title: Test E2E
    summary: Run end-to-end tests on Waldo
    steps:
    - activate-ssh-key:
        run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}'
    - git-clone:
        inputs:
        - clone_depth: "-1"
        - update_submodules: 'no'
    - recreate-user-schemes:
        inputs:
        - project_path: $BITRISE_PROJECT_PATH
    - xcode-build-for-simulator:
        inputs:
        - scheme: TravelSpot
        - configuration: Release
        - xcconfig_content: ""
    - waldo-upload:
        inputs:
        - build_path: $BITRISE_APP_DIR_PATH
        - upload_token: $WALDO_CI_TOKEN
    - nvm:
        inputs:
        - node_version: 18.20.0
    - npm:
        inputs:
        - command: install
    - set-env-var:
        inputs:
        - destination_keys: TOKEN
        - value: $WALDO_CI_TOKEN
    - set-env-var:
        inputs:
        - destination_keys: VERSION_ID
        - value: $WALDO_BUILD_ID
    - npm:
        inputs:
        - command: run ios
meta:
  bitrise.io:
    stack: osx-xcode-15.4.x
    machine_type_id: g2-m1.4core

You may notice that this Bitrise workflow is about 10 lines longer than its GitHub Actions counterpart. What I find interesting is that Bitrise provides built-in support for some tasks that GitHub Actions does not, and vice versa. Different priorities I suppose. Regardless, this Bitrise workflow is functionally equivalent to the earlier GitHub Actions workflow. Try it out!

Adding Support for Test Results

Bitrise provides a few test steps that automatically export their results to the Test Reports add-on (for example, Xcode Test for iOS and Android Unit Test). It also provides the Export test results to Test Reports add-on step which supports test results in both .xcresult and JUnit XML report format (https://devcenter.bitrise.io/en/testing/exporting-to-test-reports-from-any-step.html#using-custom-scripts-to-export-test-results). I chose to use JUnit XML report format because it is probably the most common test output format in the industry, and it is supported by virtually all testing tools (including WebDriverIO). In the case of WebDriverIO, the integrated reporter for this format is documented in https://webdriver.io/docs/junit-reporter/.

Generating Test Results

Convincing wdio to generate the report in JUnit XML format required that I add a single dependency to package.json:

  "devDependencies": {
    .
    .
    .
    "@wdio/junit-reporter": "^8.38.2",
    .
    .
    .
  },

‍

‍


Then in ios/wdio.conf.ts, I replaced this line:

  reporters: ['spec'],

with the following:

  reporters: [
    'spec',
    [
      'junit',
      {
        outputDir: './',
        outputFileFormat(options) {
          return `ios-test-results.xml`;
        },
      },
    ],
  ],

Running the Waldo Script locally, I saw that ios-test-results.xml was duly created in the top-level of my working directory. So far, so good.

Extending the Bitrise Workflow

Now that I had the test results in a format Bitrise could understand, I needed to export them to Bitrise. Fortunately, it only required adding two steps to my workflow:

    - custom-test-results-export:
        inputs:
        - search_pattern: '*/ios-test-results.xml'
        - test_name: iOS Tests
        - verbose_log: 'yes'
        - base_path: $BITRISE_SOURCE_DIR
    - deploy-to-bitrise-io: {}

All that was left to do was kick off the Bitrise workflow and see what would happen.

That’s All Folks!

Success, of course! And here are the test results as displayed on the Bitrise dashboard:

‍

All in all, quite easy.

‍