On-Device AI: What Runs on the Phone, What Cannot

Every recent flagship phone ships with a neural engine, and every mobile team is now asked some version of the same question: can this run on the device?

Sometimes the answer is yes and it is clearly the right choice. Sometimes it is technically yes and a bad idea. The distinction is worth getting right early, because it determines the architecture and it is expensive to reverse.

What the phone is genuinely good at

Modern devices handle a specific class of work extremely well: small, specialised models running on structured sensor input, repeatedly.

Speech recognition for short commands. Wake-word detection. Image classification and segmentation. Optical character recognition. Face and pose detection. Text embedding for on-device search. Translation of short strings. These are mature, the frameworks are good — Core ML on iOS, and the equivalents on Android — and the models are small enough to sit in the app bundle without anyone noticing.

Three properties make on-device the right answer for this class:

Latency. No round trip. A camera feature that classifies every frame cannot make a network call per frame, and 30ms locally is the difference between a feature that feels instant and one that feels broken.

Privacy. The data never leaves. For anything involving a camera pointed at a person, a microphone in a home, or a document containing identifiers, this is not an optimisation — it is often the only version that will pass review. We built a platform where a phone camera estimates newborn jaundice from a calibration card; that only works as a product because the image is processed where it is taken.

Availability offline. Field applications spend much of their working life without signal. A feature that stops working in a basement, a tunnel or a rural site is a feature the user learns not to rely on.

What it is not good at

The thing people usually mean by "AI" now — a large language model doing open-ended reasoning over your data — is where on-device gets oversold.

Small models will run on a phone. They will produce text. What they will not do is match what people have come to expect from a frontier model, and users do not grade on a curve for running locally. They compare it to what they used this morning.

The practical constraints stack up quickly. A useful quantised model is a substantial download, and app stores and users both notice. Sustained inference heats the device and drains the battery in a way that is very visible on a phone and much worse on older hardware, which is most hardware. Memory pressure gets the app killed in the background. And the model is frozen at ship time, so improving it means an app release and a wait for users to update — against a server-side change that reaches everyone immediately.

The division that works

In practice the architecture that holds up is neither pure. The phone does perception and filtering; the server does reasoning.

Concretely: run the small specialised model locally to turn raw sensor data into something structured and small — a transcript, a classification, a set of detected regions, an embedding. That step is fast, private and offline-capable. Then, only when the task genuinely requires open-ended reasoning, send the structured result rather than the raw input.

This is better on almost every axis. You are not uploading a video, you are uploading what was in it. The payload is small enough to queue and retry on a bad connection. The sensitive raw data never leaves. And the expensive, improvable reasoning stays somewhere you can update.

It also gives you an honest offline story. The local part keeps working with no signal, the remote part queues, and the app degrades in a way you designed rather than by throwing errors.

Ship-time realities

Three things that reliably surprise teams on the first on-device project:

Test on the oldest device you support, not the newest. Performance on current hardware tells you almost nothing about the phones most of your users actually hold. The gap is large enough to change whether a feature is viable.

Budget the battery. Inference in a tight loop will be noticed and, on iOS, surfaced to the user by name. Throttle to the frame rate you actually need rather than the one the camera offers, and stop entirely when the view is not visible.

Plan for model updates without app updates. Downloading model weights separately from the binary is more work up front and saves a great deal later — it is the difference between fixing a bad classification in a day and fixing it in a release cycle.

The short version

If the work is perception on sensor data, do it on the phone: it is faster, private and works offline. If the work is open-ended reasoning, do it on a server and send the phone's structured output rather than its raw input. The interesting engineering is in the seam between the two, not in either half.