At Measure, we build an open source mobile observability platform. One of the trickiest things to track on Android is the dreaded Application Not Responding (ANR) error. When the UI thread of an Android app is blocked for too long, Android decides to throw this error and lets the user kill the app.

This post is about how we detects ANRs and the attempts we made before getting it right.

Main thread watchdog

We started with the simplest and most well known way to detect ANRs. Run a watchdog thread that periodically posts a token to the main thread’s Handler. If the token doesn’t come back within 5 seconds, track an ANR event. The implementation was entirely in Kotlin and easy to ship.

The problem was that it was flaky in practice. The watchdog would fire on hangs that Android itself wouldn’t classify as ANRs, while missing real ANRs reported by the system.

The root issue is that Android doesn’t have a single universal ANR timeout. There are different thresholds and trigger conditions. For example, input dispatch ANRs are triggered when the main thread fails to respond to input events within 5 seconds, while broadcast receivers, services, foreground service startup, content providers and JobScheduler interactions all have their own timeout rules.

We did not end up shipping this.

Using ApplicationExitInfo

From API 30 (Android 11) onwards Android itself writes a full ANR dump automatically. ActivityManager returns a list of ApplicationExitInfo records describing the app’s recent process exits including ones with REASON_ANR. It also contains the state of every thread at the time of crash.

However, this API has a few limitations.

First, it’s API 30 and above only. We support older releases (API level 21 and above) where ApplicationExitInfo doesn’t exist. We needed ANR detection that works on every device our SDK runs on.

Second, it only tells you about process exits after the fact. We only see the record on the next app launch, by which point everything we’d have wanted from the moment of the ANR (for example a screenshot of the moment the ANR occurred) is gone with the process.

Third, the system keeps these records in a bounded ring buffer, and older entries get evicted as new ones come in. There’s no guarantee the specific ANR we want is still around when the app is launched again.

We use ApplicationExitInfo where it’s available, but cannot fully rely on it to achieve our goals.

The real signal

Signals are how Unix-style operating systems poke a process when something asynchronous happens. Pressing Ctrl+C in a terminal sends SIGINT to the foreground process. A segmentation fault generates SIGSEGV. Each signal has a number (SIGINT is 2, SIGKILL is 9, SIGQUIT is 3) and a default behavior the kernel applies if the process doesn’t override it.

For SIGQUIT, the default on Linux is to terminate the process and write a core dump. Android overrides this behavior. It shows the “App Not Responding” dialog and writes an ANR report.

All we needed was a way to intercept this signal, record an ANR and pass it back to the system to continue doing its thing. It was harder to do than we initially thought.

Catching SIGQUIT

The obvious first move to detect a signal is to register a signal handler. So we registered one to handle SIGQUIT.

struct sigaction sa = { .sa_handler = on_sigquit };
sigaction(SIGQUIT, &sa, NULL);

In a regular Linux process this is enough to get notified of the signal.

On Android the handler never runs.

To see why, it helps to know there are two ways a thread can deal with an incoming signal.

The first is what we just did. Install a handler with sigaction, and when the signal arrives the kernel pauses the thread mid-instruction, runs the handler, and resumes. The catch is that the handler runs in interrupted context. You can’t allocate, take a mutex or call into the JVM. The list of things you can safely do (the async-signal-safe list) is short.

The second approach is to block the signal on every thread, then dedicate one thread to pulling it off the pending queue with sigwait or sigwaitinfo. The signal arrives as a return value rather than an interrupt, so the dedicated thread runs in ordinary context and can allocate, take locks, and call into the runtime.

Android picks the second pattern for SIGQUIT. At runtime startup, it blocks SIGQUIT in every thread and spawns a dedicated thread named Signal Catcher that sits in a loop calling sigwaitinfo. When SIGQUIT arrives, no thread has it unblocked, so the kernel has nothing to interrupt. The signal sits in the pending queue until Signal Catcher pulls it out to produce the ANR trace.

That’s why our handler never fired. Installing a handler doesn’t unblock the signal, and every thread inherited SIGQUIT blocked at runtime startup. The kernel had no thread to interrupt, so the signal queued, and Signal Catcher continued it’s work.

Watchdog 2.0

To get our handler running, we need our own thread in the process with SIGQUIT unblocked. Then SIGQUIT becomes deliverable, the kernel picks our thread, and the handler runs.

We spawn one thread, call it Watchdog, and unblock SIGQUIT for it with pthread_sigmask. Signal masks are per-thread, so Signal Catcher is unaffected. Watchdog is now the only thread in the process where SIGQUIT is unblocked, which makes it the only thread the kernel can deliver to.

The handler waits for the signal to arrives.

static void on_sigquit(int sig) {
    sem_post(&anr_sem);
}

The trick is to keep the handler as small as possible and have Watchdog do the real work after the handler returns. A semaphore makes the handoff.

Watchdog spends most of its life waiting on the semaphore. When SIGQUIT arrives, the kernel runs our handler on Watchdog. The handler wakes the semaphore and returns. That’s all it does, because waking a semaphore is one of the few things you can safely do from inside a signal handler.

Watchdog is now back in its own code. The handler ran on this thread, did one async-signal-safe thing, and returned. Everything past the semaphore wait is ordinary thread code, so Watchdog can take locks, allocate, call into the JVM, walk threads and capture the state we want for the ANR.

This works, but now it breaks the platform’s ANR flow.

Signal Catcher is still parked in sigwaitinfo, waiting on a signal we just consumed. No SIGQUIT means no “App Not Responding” dialog.

Handing the signal back

Watchdog needs to send a fresh SIGQUIT to Signal Catcher so the platform machinery continues to run.

We can’t just send another SIGQUIT to the process. The kernel looks for a thread with SIGQUIT unblocked, finds Watchdog (still the only one), and the signal comes right back to us.

We need to target Signal Catcher directly. The primitive for that is tgkill, which delivers a signal to a specific thread by its TID. Which we don’t have.

Getting Signal Catcher’s TID is the awkward part. There’s no API for it, but /proc/self/task/ has a directory for every thread in the process, and each directory has a comm file with the thread’s name. At SDK init we walk the directory once, find the entry that reads “Signal Catcher”, and grab its TID. When an ANR fires we record it and send SIGQUIT back to Signal Catcher via tgkill.

The platform’s ANR flow now runs as it would have without us in the picture, except we now have our own data captured at the moment it happened.

Putting ANRs on the timeline

Capturing an ANR is complex, but fixing them is even harder. We did all of this so that every ANR comes with the full picture of what led up to it.

First, a timeline of events that occurred before the ANR was triggered: HTTP requests, navigation transitions, lifecycle callbacks, gesture events and custom events.

Second, the ApplicationExitInfo record from Android along with the stack trace it provides. Third, an optional screenshot of the screen at the moment the ANR fired.

This allows scrolling back through the session to see what the user was doing right up to the moment the app froze.

You can interact with a real session timeline at https://measure.sh/product/session-timelines.

Sources

The native ANR detection code lives in our Android SDK on GitHub. On the platform side, AOSP’s signal_catcher.cc is the file that implements the Signal Handler thread.

The man pages for the functions and signals mentioned above are linked below for reference.

• signal(7)

• sigaction(2)

• sigwait(3)

• sigwaitinfo(2)

• pthread_sigmask(3)

• sem_post(3)

• tgkill(2)

• proc(5)

Most observability platforms started life watching servers. They got really good at it. Tracing requests across microservices, tracking error rates per endpoint and alerting on P99 latency spikes. Mobile came later and the same architecture and data models were extended to support it.

Except mobile observability is a fundamentally different problem. The failure modes are different, the constraints are different, and if you care about monitoring your app well, it’s worth understanding how and why that matters.

You don’t own the machine

When a backend service misbehaves, you SSH in, read the logs, bump the memory, restart the process. You have full control.

On mobile, your code runs on a device in someone’s pocket, on a network you’ve never heard of, in a country you’ve never been to, on an OS version you didn’t know any one was still using.

That device might have 2 GB of RAM shared across 40 apps. It might be running Android 9 with a manufacturer skin that patches the lifecycle callbacks differently. It might be an iPhone SE on a train going through a tunnel.

This changes how you collect data, what data you collect, and what you do with it. Every byte of telemetry has to survive unreliable networks, respect battery life, and fit through bandwidth constraints, all while making sure your observability tool doesn’t itself become a performance problem.

You can’t revert an app release

Backend deploys are reversible. Something breaks, you roll back or quickly patch a fix, the whole cycle ideally takes minutes. You can ship ten times a day.

Mobile releases go through app store review. That’s hours at best, days at worst. And even after your fix is approved, users have to actually update. Some won’t for weeks. Some never will.

The life of a bug that ships is just dramatically higher. A backend bug is a bad hour. A mobile bug can be a bad week or even a month. You need to spot the problem forming early in a release cycle while your rollout is still small or pay a higher price later.

Crashes aren’t errors

A 500 error on a server is bad, but the server restarts or just keeps running. The next request probably works fine.

A crash kills the session. The user was in the middle of something (placing an order, writing a message) and now they’re staring at their home screen. There’s no automatic retry. There’s just a person deciding whether your app is worth opening again.

Then there are ANRs (Application Not Responding). The app hasn’t crashed, it’s technically still alive, but it’s frozen and the OS is asking the user if they want to force close. There’s really no backend equivalent. It’s one of the most frustrating experiences a mobile user can have, and a lot of observability tools don’t even track it properly.

Stack traces are useless without symbolication

When you ship a mobile app, you don’t ship the code you wrote. Release builds go through optimisation and obfuscation - ProGuard or R8 on Android, symbol stripping on iOS. Your carefully named PaymentProcessor.processTransaction() becomes something like a.b.c() on Android or a hex memory address on iOS.

This is good for app size and security. It’s terrible for debugging. When a crash comes in from production, the stack trace is gibberish. A wall of obfuscated class names and stripped addresses that tells you nothing about what actually went wrong.

To make it readable again, you need symbolication: mapping those mangled names and addresses back to your original source code. On iOS, that means dSYM files generated at build time. On Android, it’s ProGuard or R8 mapping files. Every build produces its own mapping, and if you lose it or upload the wrong one, your crash reports are permanently unreadable for that version.

This is an operational burden that just doesn’t exist in backend. Your server logs say NullPointerException in PaymentService.java:142 and you go fix it. For mobile, you need a pipeline that automatically captures mapping files for every build, matches them to the right app version, and symbolicates crash reports as they come in. Get any step wrong and you’re staring at 0x0000000100a3b2c4 wondering what went sideways.

It’s one of those things that’s invisible when it works and completely debilitating when it doesn’t.

A request is not a session

Backend observability is built around the request. A request comes in, gets traced across services, produces a response. Clean and well-bounded.

Mobile users don’t make requests. They have sessions. They open the app, tap around, switch to another app, come back twenty minutes later, scroll, hit a button, get interrupted by a phone call, return, and eventually close the app. Or don’t, it just gets killed by the OS when memory runs low.

Understanding what went wrong means reconstructing that journey. What screens did they visit? What did they tap? What network calls fired? What was the memory pressure at the time? What did the screen actually look like right before the crash?

A timestamped error log tells you almost nothing. You need the full session timeline: navigation events, gestures, network calls, resource usage, UI state and much more context to see what actually happened.

Performance is relative

When a backend engineer talks about performance, they mean latency and throughput on known hardware. You know exactly what you’re working with.

Mobile performance might mean cold start time, warm start time, time to first frame, frame rendering jank, memory consumption, battery drain, or app size. And every one of these varies across devices. Your app starts in 400ms on a Pixel 9 and 4 seconds on a budget Samsung from 2020. Both are real users.

If your observability tool only shows you averages, you’re seeing a number that represents nobody. You need to slice by device, OS version, app version, network type, geography. You need the distribution, not the mean.

The latency you don’t see

Backend tracing follows a request across services. A request comes in, hops through some microservices, and produces a response. The trace has a clear start and end.

Mobile traces are messier. A trace might span multiple screens as a user works through a flow. Adding items to a cart, entering an address, hitting checkout. They might pause halfway through to reply to a text, or lose connectivity on the subway, or get a phone call. The app backgrounds, the OS might reclaim memory, and the user may or may not come back.

Then there’s the network side. Your server dashboard says the API responded in 200ms but the user waited three seconds. The gap is everything that happened before the request reached your server: DNS resolution on a flaky network, TLS handshake on a slow connection, request queuing while the cellular radio wakes up.

Backend traces pick up at the API gateway. Everything before that is invisible unless your mobile tooling captures it.

Mobile-aware tracing connects both sides, the on-device spans and the backend spans, so when a user says “the app felt slow,” you can actually tell whether the problem was the network, the client, or your API.

The telemetry paradox

A hard part of mobile observability it that the thing you’re measuring is the thing being affected by the measurement.

Every event you log takes CPU, memory, and battery. Every network request to ship telemetry uses bandwidth the user might be paying for. A heavy SDK that captures everything will make the app slower and drain more battery, creating the exact problems you’re trying to detect. It doesn’t matter much if a tracing sidecar uses an extra 200MB of RAM on a server. On a phone, your SDK’s overhead is a direct tax on user experience.

Mobile SDKs have to be absurdly efficient. Batch intelligently, compress aggressively, back off when resources are tight. Capture enough to be useful but little enough to be invisible.

The fragmentation nightmare

“Works on my device” is the mobile version of “works on my machine,” except it’s orders of magnitude worse.

There are thousands of distinct Android devices in active use — different screen sizes, chipsets, GPU capabilities, RAM configurations, manufacturer customizations, OS forks. iOS is more constrained but still spans multiple hardware generations and OS versions.

A bug might only reproduce on Samsung devices running Android 12 with a specific GPU driver. Or on iPhone SE in low power mode. Or only when the app is restored from background after 30 minutes on a slow network.

You see a 0.5% crash rate and shrug, but that might be 100% of users on a specific device having an awful time.

A different beast

Mobile observability isn’t backend observability with a different client library.

It’s a related but different discipline with it’s own primitives and fundamentally different failure modes.

The tooling that your mobile team depends on should recognise and respect that.