Opening article
RedNote’s 3B‑parameter OCR model dots.ocr has outperformed Gemini 2.5 Pro on OmniDocBench, demonstrating that large‑scale OCR can run on Apple devices. The team converted the model to Core ML and MLX, leveraging Apple’s Neural Engine for power efficiency. The conversion involved simplifying attention mechanisms, removing variable‑length masking, and addressing dynamic control flow issues. The resulting Core ML model is over 5 GB and takes more than a second for a single forward pass of the vision encoder, highlighting the challenges of on‑device deployment.
Every year our hardware is a little more powerful, our models a little smarter for each parameter. In 2025, it is more feasible than ever to run truly competitive models on-device. dots.ocr , a 3B parameter OCR model from RedNote, surpasses Gemini 2.5 Pro in OmniDocBench , making OCR a truly no compromises on-device use case. Running models on-device is certainly appealing to developers: no smuggling API keys, zero cost, and no network required. However, if we want these models to run on-device, we need to be mindful of the limited compute and power budgets.
Enter the Neural Engine, Apple's custom AI accelerator that has shipped with every Apple device since 2017. This accelerator is designed for high performance whilst sipping battery power. Some of our testing has found the Neural Engine to be 12x more power efficient than CPU , and 4x more power efficient than GPU .
Whilst this all sounds very appealing, unfortunately the Neural Engine is only accessible through Core ML , Apple's closed source ML framework. Furthermore, even just converting a model from PyTorch to Core ML can present some challenges, and without a preconverted model or some knowledge of the sharp edges it can be arduous for developers. Luckily, Apple also offers MLX , a more modern and flexible ML framework that targets the GPU (not the Neural Engine), and can be used in conjunction with Core ML.
Dots.OCR consists of two key components: A 1.2B parameter vision encoder trained from scratch, based on the NaViT architecture , and a Qwen2.5-1.5B backbone. We will be using CoreML to run the vision encoder, and MLX to run the LM backbone.
This is again due to the masking logic to handle variable length sequences. Since we are only processing a single image (not a video or batch of images), we don't really need attention masking at all! Therefore, we can just use a mask of all True . To prepare ourselves for the Neural Engine conversion, we also switch from using a boolean mask to a float mask of all zeros, as the Neural Engine does not support bool tensors commit here
In this three part series, we will provide a reasoning trace of how we converted dots.ocr to run on-device, using a combination of CoreML and MLX . This process should be applicable to many other models, and we hope that this will help highlight the ideas and tools needed for developers looking to run their own models on-device.
To follow along, clone the repo . You'll need uv and hf installed to run the setup command:
If you just want to skip ahead and use the converted model, you can download it here .
Demonstrating that state‑of‑the‑art OCR can run on‑device shows the feasibility of privacy‑preserving, low‑latency AI applications. It also highlights the need for efficient model conversion and optimization tools for mobile hardware.