Testing private APIs

If you’re starting to explore options for cloud testing your mobile apps, some of your first questions might be: is your app safe to test in production? Those of you deploying enterprise apps via MDM might wonder, how would a cloud-run test access the app’s backend, which is accessible only within a corporate network or via a VPN? Or how do you test features that may only be available in UAT or staging environments?

Maybe you’re farther along, experimenting with cloud testing, but are wondering how you can fix unreliable tests? Or how you can automate completing flows that have undesirable side effects like hailing taxis, ordering food or making payments. Or maybe you want to know how to avoid bothering ops with patches to production data just to keep the tests happy.

Many of these questions have similar answers, and the approaches below will result in safer, more robust tests that can be run in the cloud, QA, or by developers. The bad news is that there’s no free lunch: your tests will never be perfect, and trade-offs need to be made.

Approaches

One of the solutions to many of these problem is to remove network access from the application under test entirely. That is, instead of making API calls to your live backend service, have the app substitute canned responses instead. This approach is sometimes called faking or stubbing.

Another solution would be to deploy a separate, isolated instance of the API. This instance could be made accessible just to the cloud testing service, and loaded with data crafted by either developers or QA to facilitate ease of testing.

One last solution would be to punch a hole into your private network from the outside, and lock down the test app and firewall to minimize risk.

So let’s dive into how we might apply each of these approaches, from the perspective of somebody who is starting with an Android app and wants to produce an APK that can be handed off to a cloud service such as Waldo for testing. Let’s use android-gif-example as our app, as it serves as a clean, modern example.

Faking or stubbing an API

The first step is to identify the APIs that you wish to mock, and either isolate them behind an interface or class that can be cleanly substituted by a fake. If your application doesn’t have a clean interface over the API layer such as this, some degree of refactoring will be necessary — which can be a worthwhile exercise in and of itself.

Creating a test variant

Since the ultimate goal is to produce an APK with faked out API calls, we will create a new Android _variant_ for our build, by means of two new _product flavors_. Then we will supply the real API calls in a _live_ flavor, and faked ones in a _mock_ flavor. Modify your build script (usually `app/build.gradle`, or in our example app, `app/build.gradle.kts`) to include a new product flavor for the test environment, tied together by a _dimension_ called `api`.

productFlavors {
   flavorDimensions("api")
   productFlavors {
       create("live") {
           setDimension("api")
       }
       create("mock") {
           setDimension("api")
       }
   }
}

Assuming no other dimensions, Gradle will now produce four variants of your APK:

• debugLive
• debugMock
• releaseLive
• releaseMock

`releaseLive` is what you ship to end users, `debugLive` is what developers would work with, and `debugMock` or `releaseMock` would be what you test with.

Substituting the faked API

Right now the live and mock APKs are identical. The next step is to use a little known feature of variants that allows us to substitute different implementations of a class in each. Since `RiffsyModule` contains a Dagger module that supplies the live API Retrofit object, it is a good place to substitute in a fake. We accomplish this by moving the existing API code into the source directory of the _live_ variant:

$ mkdir -p app/src/live/java/com/burrowsapps/example/gif/di/module
$ git mv app/src/main/java/com/burrowsapps/example/gif/di/module/RiffsyModule.kt app/src/live/java/com/burrowsapps/example/gif/di/module

Note how the directory `app/src/live` is named after the _live_ flavor: code in this directory is only included in the `liveDebug` and `liveRelease` variants. Similarly, we provide a new, faked implementation of the API via a module of the same name and package for the _mock_ flavor:

(in app/src/mock/java/com/burrowsapps/example/gif/di/module/RiffsyModule.kt)
...
@Module
class RiffsyModule() {
 @Provides fun provideRiffsyApi(): RiffsyApiClient = object : RiffsyApiClient {
   override fun getTrendingResults(...) = ...
   override fun getSearchResults(...) = ...
 }
}

Now mock flavor APKs will use the fake data supplied by the `RiffsyApiClient` in this class. If you have familiarity with Dagger or Retrofit, you might be thinking that you could avoid this with a build-time flag and some clever dependency injection, and you’d be right. But this class substitution approach is more broadly applicable to the sorts of code you’d find in legacy projects or apps with tightly-coupled components.

Further techniques

This faked API approach lends itself to a number of further techniques: The fake could be stateful, and remember data from API calls that would normally modify data on the server. For example, in a todo list app, you could maintain a `MutableList<Todo>` in the app itself that backs the CRUD operations of your API. Conveniently, because this state is in-memory, it disappears between runs of the app, improving test repeatability.

The fake could respond differently depending on the test user that logs in. One could be rigged to trigger errors, be sluggish to return data, or receive rare responses on certain screens, making it easy for a cloud test to hit otherwise inaccessible parts of the app. In this way, tests against faked data can be more comprehensive than live data.

Finally, note that if you are using **OkHttp**, it includes some components like `MockWebServer` that make it easy to return fake responses from Retrofit services.

Disadvantages

There are some drawbacks with this approach. Your networking code won’t be exercised at all, and any misalignment between the deserialization code on the client and the formats provided by the server won’t be caught. Further, this APK cannot be used for Waldo’s API monitoring, as it would be monitoring fake responses.

To mitigate this risk, consider writing a separate suite of integration tests (using AndroidX for example) that run against a live backend (so staging, UAT, or prod) within your network and can check integration between your API code and live services. If you have isolated your API code for purposes of faking those calls, then you can simply test the live flavor of that class in isolation.

You can also run tests against both flavors as if they were separate apps: the live APK with some basic smoke tests against critical flows like sign-up, and the mock APK for more comprehensive testing.

Deploying an integration API instance

There are some situations where the mock approach can be more trouble than its worth. Say your API is complex, or tightly coupled to the app, making isolation and refactoring a challenge. Perhaps you’ve decided that there is unacceptable risk in testing and deploying significantly different builds.

One way to preserve most of the benefits with mocking while keeping the codebase unified would be to stand up a continuous integration (CI) instance of the API. While there is a huge up-front cost, the CI instance can then be loaded with static set of fake data and accounts, tailored to the needs of individual test cases. If tests are generating garbage data each run, it is much easier to simply restore this static data than carefully patching prod.

This test environment can also be isolated from other production systems by running it in a separate VM or kubernetes cluster, and introducing separate databases, etc. The CI API can be exposed via an IP that whitelists only the testing service, so there’s little risk of intruders gaining access to anything but the test environment.

Injecting test API URLs

Producing an APK that uses the testing endpoints is simple. First, we’ll introduce build-time constants into two new flavors for the endpoints that differ, calling them _prod_ and _ci_. Using the example app from earlier, we make a similar change to the build file:

productFlavors {
     flavorDimensions("env")
     productFlavors {
       create("prod") {
         setDimension("env")
         buildConfigField("String", "BASE_URL", "\"https://api.riffsy.com\"")
       }
       create("ci") {
         setDimension("env")
         buildConfigField("String", "BASE_URL", "\"https://api-ci.riffsy.com\"")
       }
     }
   }

In the example app, the URL parameter from the build script will be available in `BuildConfig`, and so imported and used as follows:

import com.burrowsapps.example.gif.BuildConfig
   ...

   @Module
   class RiffsyModule(private var baseUrl: String = BuildConfig.BASE_URL) {
    ... (API access code doesn't differ between environments) ...
   }

As you can guess, the resulting APKs will be identical except for this string.

Disadvantages

The operational burden of creating and maintaining the external CI environment is a large disadvantage, and you will have to keep it up to date or risk your tests not being reflective of the true production environment.

Another disadvantage is security: while production is not at least, attackers may be able to learn about your internal services by exploring a test environment. For added security, you could introduce an authorization token that all requests must possess, otherwise they are rejected. It is generally easy to configure points of ingress to require such a token, and adding it to all requests is simple with Retrofit. Just add the token to the _ci_ flavor from earlier:

create("prod") {
     ...
     buildConfigField("String", "CI_AUTH_TOKEN", "null")
   }
   create("ci") {
     ...
     buildConfigField("String", "CI_AUTH_TOKEN", "\"bWFkZSB5b3UgbG9vawo=\"")
   }

And then use an _OkHttp interceptor_ to include the token, only when present, in all requests:

(method of app/src/main/java/com/burrowsapps/example/gif/di/module/NetModule.kt)

internal fun createOkHttpClient(...): OkHttpClient = OkHttpClient.Builder()
  .addInterceptor(object :Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
      return if (BuildConfig.CI_AUTH_TOKEN != null) {
        chain.proceed(chain.request().newBuilder()
          .addHeader("Authorization", "Bearer ${BuildConfig.CI_AUTH_TOKEN}").build())
      } else {
        chain.proceed(chain.request())
      }
    }
  })
  ...

Note that the token is embedded in the build script for simplicity: you’ll likely want to extract it into an environment variable or Gradle property only present on your build host.

Testing in production after all

Perhaps the above are too much effort to gamble on a testing experiment that may not bear fruit. Fair enough, and after all, the lowest-effort solution is to simply use the app and backend you already have.

However, if you must still keep your servers private, you might consider setting up a proxy server to let some traffic sneak in. Often proxies can also be configured to only allow access to select internal hosts. It is also much easier to bundle the proxy configuration in an APK, and many mobile HTTP libraries provide a facility for doing this.

This approach has the advantage of also being reasonably secure if your proxy server only allows HTTPS, requires authentication, and you whitelist the testing service’s IP address. It is also better than a reverse-proxy solution because you don’t need to explicitly setup individual ports to route traffic to the appropriate backend services: there is one single point of entry for all traffic.

Configuring your APK to proxy traffic

Since setting up a proxy server is beyond the scope of this article, let’s assume you’ve got a SOCKS proxy with authentication enabled. We turn again to our example app to see how we might proxy our API calls. As with the other approaches, we use two build flavors, _live_ and _ext_ (for external), for distribution and cloud testing, respectively.

productFlavors {
     flavorDimensions("env")
     productFlavors {
       create("prod") {
         setDimension("env")
         buildConfigField("String", "PROXY_HOST", "null")
         buildConfigField("int", "PROXY_PORT", "0")
         buildConfigField("String", "PROXY_AUTHENTICATOR", "null")
       }
       create("ext") {
         setDimension("env")
         buildConfigField("String", "PROXY_HOST", "proxy.mycompany.test")
         buildConfigField("int", "PROXY_PORT", "1080")
         buildConfigField("String", "PROXY_AUTHENTICATOR", "\"bWFkZSB5b3UgbG9vawo=\"")
       }
     }
   }

We also need to modify our CI variant’s **OkHttp** setup to use a proxy, and also authenticate proxy requests. This time, we edit `NetModule.kt` as it contains the `OkHttpClient` configuration:

(methods of app/src/main/java/com/burrowsapps/example/gif/di/module/NetModule.kt)

   internal fun createOkHttpClient(
     application: Application
   ): OkHttpClient = createOkHttpClientBuilder()
     .addInterceptor(createHttpLoggingInterceptor())
     .connectTimeout(CLIENT_TIME_OUT, TimeUnit.SECONDS)
     .writeTimeout(CLIENT_TIME_OUT, TimeUnit.SECONDS)
     .readTimeout(CLIENT_TIME_OUT, TimeUnit.SECONDS)
     .cache(createCache(application))
     .build()

   private fun createOkHttpClientBuilder(): OkHttpClient.Builder {
     if (BuildConfig.PROXY_HOST == null) {
       return OkHttpClient.Builder()
     }

     return OkHttpClient.Builder()
       .proxy(Proxy(Proxy.Type.SOCKS, InetSocketAddress(BuildConfig.PROXY_HOST, BuildConfig.PROXY_PORT)))
       .proxyAuthenticator(object : okhttp3.Authenticator {
         override fun authenticate(route: Route?, response: Response): Request? {
           return response.request.newBuilder()
             .header("Proxy-Authorization", BuildConfig.PROXY_AUTHENTICATOR)
             .build()
         }
       })
   }

So for _ext_-flavored APKs, we’ll use the proxy server to securely connect to the API.

Disadvantages

In my opinion, running tests against production is dicey at the best of times. You’ll need to roll new features out to prod in order to run tests against them, and if your goal is to catch regressions or track the app’s UX over time, testing against production adds little except risk.

Conclusion

As we’ve seen, if you’re serious about testing and want to take cloud continuous integration for a spin, private networks or finicky APIs don’t have to be showstoppers. And in some cases, reevaluating the structure of your code and refactoring to isolate network dependencies can have other benefits, such as improving test robustness and exhaustiveness. Learn more here.