Running a neural network across platforms

The problem: same app idea, different hardware

On iPhone, models typically run through Core ML, Metal/GPU and the Apple Neural Engine. On Android, acceleration depends on the device: Qualcomm Snapdragon can use the Hexagon NPU and Adreno GPU, while other vendors ship their own NPUs and drivers. Add CPU fallbacks, differing Android versions and various runtime APIs.

An app should therefore not assume that "NPU" means the same thing everywhere. A stable app-level interface is better: predict(strokes) -> topK labels. Underneath, the implementation may differ per platform.

Recommended strategy for a small QuickDraw CNN

For DrawLa the task is manageable: strokes are rasterised into a 1 × 96 × 96 image, a CNN computes 345 logits, and the app shows the most likely words. That model is small enough to run locally on both mobile platforms.

PyTorch training checkpoint
  → neutral export: ONNX
  → iOS: Core ML / .mlpackage
  → Android: LiteRT or ONNX Runtime Mobile
  → shared app API: predict(strokes)

The core stays identical: the model was trained and tested once. For distribution, however, two or more optimised artefacts are produced. That is normal, and usually better than a minimal solution that runs on both platforms without using either well.

Option A: ONNX Runtime Mobile as the common denominator

ONNX Runtime Mobile executes ONNX models on iOS and Android alike. Attractive if you want to keep one model format. For a small CNN, ONNX is especially natural because the operators are simple: convolutions, normalisation, activation, pooling and a linear layer.

Upside: one neutral export path out of PyTorch, one runtime family for both platforms. Downside: maximum hardware acceleration and binary size still have to be tested per platform.

Source: ONNX Runtime Mobile.

Option B: native artefacts for the best platform fit

On iPhone, Core ML is the native route. With coremltools, PyTorch models convert directly into Core ML formats. Apple can then deploy CPU, GPU and Neural Engine as appropriate, provided the model and its operators are suitable.

On Android, LiteRT or ONNX Runtime Mobile are the usual paths; for Qualcomm devices, additional SDKs or delegates may be interesting. This variant produces several artefacts but is often the most robust performance strategy in production apps.

Sources: Core ML Tools, LiteRT.

Option C: ExecuTorch for PyTorch-native deployments

ExecuTorch is PyTorch's own route to on-device AI. It targets phones, embedded systems and edge devices, and can appeal to teams that want to keep training, export and deployment inside the PyTorch ecosystem for as long as possible.

For DrawLa, ExecuTorch would be technically plausible, but you would have to weigh the overhead against ONNX Runtime Mobile or Core ML. With a small CNN, the simpler runtime is often the better choice.

Source: ExecuTorch documentation.

The real problem is not the model

Anyone planning such a port regularly underestimates where the work is. Converting the model is one call to coremltools or an export to ONNX. Preprocessing is the hard part.

In DrawLa the rasteriser currently lives in Python: determine the bounding box, centre, scale with an eight per cent margin, draw segments two pixels wide. On device, the same procedure would have to exist in Dart, Swift or Kotlin — and be pixel-identical.

This is exactly where errors nobody notices are born. If one language rounds differently, a margin is off by a pixel, or line drawing interpolates slightly differently, the model receives inputs it never saw during training. It does not crash and reports no error — it simply gets worse. Without a comparison against the server implementation, that can go unnoticed for months.

Rule of thumb: for every line of model conversion, budget ten lines of preprocessing tests.

How to guarantee parity

The tool for this is reference fixtures. Pick a handful of drawings, store the server-rasterised image and the resulting probabilities for each, and ship both as test data with the app.

fixtures/
  cat_01.strokes.json      # input
  cat_01.raster.png        # expected 96×96 image
  cat_01.logits.json       # expected output

The test on each platform then checks two things separately: does the rasterised image match the reference exactly? And are the outputs within a small tolerance — somewhat more generous for quantised models than for fp32?

The separation matters. If only the output differs, model conversion or quantisation is responsible. If the image already differs, preprocessing is at fault — and any search inside the model is wasted time.

The Flutter detour

DrawLa is written in Flutter, which adds a layer usually missing from platform-neutral discussions: Dart cannot address Core ML or LiteRT directly. The call goes either through a platform channel into native Swift or Kotlin code, or via FFI into a C library.

Both have a cost. A platform channel serialises the data on every call — not dramatic at eight times per second, but not nothing either, and it means maintaining native code for two platforms. FFI avoids the serialisation but demands a C interface and careful memory handling.

For a Flutter app this shifts the balance noticeably towards ONNX Runtime Mobile: one model format, one integration, one set of tests. The last percentage point of hardware acceleration that Core ML would additionally extract does not justify double the integration effort at a model size of two megabytes.

A pragmatic target architecture

The cleanest architecture separates three things: training, model artefacts and the app interface. Training stays in PyTorch. The export produces tested artefacts for iOS and Android. The app itself knows only a small inference interface and need not care whether Core ML, ONNX Runtime or LiteRT sits underneath.

shared Dart/Swift/Kotlin API
  rasterize(strokes) -> Float32/UInt8 tensor
  predict(tensor) -> topK labels

iOS backend
  Core ML model

Android backend
  ONNX Runtime Mobile or LiteRT model

This keeps the product logic cross-platform while each platform uses its best inference path. For fast sketch recognition that usually matters more than a single framework that can do everything on paper.

Whether the move on device is worth it at all is discussed in Mobile & Edge AI; the model itself is described in QuickDraw with PyTorch.