Picking a crash reporting tool

One of the most challenging tasks in mobile app development is reproducing issues users are hitting in production. Several factors come together to frustrate your investigation: users might only hit bugs on uncommon devices, or only when they are several OS patches or app versions behind the latest and greatest. Especially on Android, market fragmentation means some users may never even get the latest, so ignoring bugs on lagging devices is rarely an option.

A dedicated QA team can help with this, but even then there can be a disconnect between their testing efforts and the developers that need to fix those bugs. QA might be able to reproduce issues and record how they did it, but those steps might lack key context, or don’t work for developers who work in an emulator or on a cutting-edge device. Due to the rise of remote work, developers may also be physically separated from the team and not have access to test devices at all.

Sometimes the app isn’t outright crashing, but instead exhibits unusual behavior like empty screens or vague error dialogs. Developers may be able to speculate on causes, but without hard evidence, they’re making blind fixes, and won’t find out if they’re right until the next release.

Below, a review of four popular crash reporting tools.

The solutions

Reproducing a user’s exact setup is likely to be impossible in many cases — so the only solution is to go to them. Since arranging flights to frustrated users likely isn’t in the budget, the only practical solution is to remotely and automatically gather crash and contextual data from users’ devices to hand off to devs and testers.

So what specific data do we want? The most useful clues for debugging include:

Traces: Most fatal (i.e. app-closing) errors generate a stacktrace through the code at the point they occurred. Often this is the starting point for the investigation, but often this gives only the symptom instead of the true cause.

Logs: Developers often write unexpected events or errors to logs, which provide a trail of activity leading up to a crash.

Navigation: Users rarely follow the happy path — knowing which one the user took through the app can lead to insights.

Environment: Which device model are they using? Which OS and app versions are they using?

Gathering this data on the device is easy enough, but then where do we upload it? The good news is that the mobile crash reporting space is crowded with competition, with dozens of packages out there to fit every organization.

Picking a crash reporting tool

At their core, these tools are mostly interchangeable: they provide *integrations* with various platforms (Android, iOS, backend frameworks, etc.), catch and deobfuscate traces, capture environment info and app state, and phone home with fatal crashes. So keep in mind the differentiators here are mainly what views they give you on those crashes, how they are deployed, and advanced features.

Making a catch-all recommendation is impossible, but there are a few stand out packages depending on your needs:

Firebase Crashlytics

A solid offering from Google integrated with Firebase, and the majority choice of app developers.

Strengths: Large user base and community suggests it will be around for awhile, can likely handle any exception volume, free to use at any scale.

Weaknesses: No self-hosted option, potential security concerns with handing app and user data to Google, not as full-featured as a dedicated solution, not completely decoupled from Fabric, which includes things you may not need.

Recommendation: Pick for the rare combination of quick, easy and free, if the basic features cover your needs and you prefer a cloud solution.

Sentry

A popular open-source package focused on error reporting.

Strengths: Free/open-source version is great for DIYers^1 , dozens of platforms and frameworks are covered by first-party and community-made integrations, powerful reporting coming soon in version 10.

Weaknesses: Advanced features such as tracing require significant developer investment, no paid support option for on-premise deployment, and SaaS pricing is pay-by-event, which could get expensive depending on anticipated volumes and incentivizes you to collect less data.

Recommendation: Pick if you want a popular option, and don’t mind an additional investment of setup time.

^1 Released under a controversial Business Source License. While we aren’t lawyers, this is unlikely to be an issue unless you’re a FoSS diehard or if crash reporting is part of your product.

Bugsnag

A smaller, closed-source player, Bugsnag makes it easy to get up and running quickly.

Strengths: Paid on-premise option, pay per login provides excellent value for small teams, mobile integration collects more out of the box, and scales well with large error volumes.

Weaknesses: Pay-per-user model could be expensive for larger organizations, weak free tier offering, and being less popular means leaning on paid support rather than the community.

Recommendation: Pick if you have a small team or prefer to pay in cash rather than time.

Countly

Combines deep analytics with competent crash reporting.

Strengths: Bundles powerful analytics into the same platform as crash reports, paid on-premise option, and free community/open-source version with permissive AGPL licensing.

Weaknesses: Struggles with high error volumes, and bug reporting isn’t as well-developed as focused products like Sentry or Bugsnag.

Recommendation: Pick if you want one solution for all your analytics and error reporting.

Conclusion

Mobile app development presents unique challenges in chasing down bugs, because users are bringing one of dozens of possible iOS devices, or one of thousands of Android devices. Multiple major versions of each mobile OS are in widespread use at any given time. So collecting, aggregating and reporting on crashes by platform and version is a necessity.

Your crash reporting infrastructure becomes a valuable tool for internal staff: where app crashes can be triaged, and detailed QA or CI crash reports are organized for when bugs land on developers’ desks.

When combined with physical device testing, you’ve got an excellent start at reproducing fringe issues that otherwise lead to frustrated users and one-star reviews.

Now then, why not try reproducing those tricky issues with Waldo?

Psst!
In our next post, we’ll dig into the details of integration using the Android example app from our previous article, combined with the Sentry open-source platform. Make sure to subscribe to our blog so you don’t miss it!

‍