Opening article
The Hugging Face blog "LLM Inference on Edge" explains how to build a React Native mobile app that runs large language models locally on a phone. It highlights that newer models, such as DeepSeek R1 Distil Qwen 2.5 with 1.5 billion parameters, are small enough to fit on mobile devices. The tutorial uses the llama.rn binding for llama.cpp to load GGUF files efficiently, and it relies on the EdgeLLM repository for full source code. The guide covers environment setup, including Node.js, react‑native‑community/cli, and Android Studio emulators, and walks through fetching and downloading quantized GGUF models from the Hugging Face Hub using axios. The app’s UI is built with React Native components such as SafeAreaView, ScrollView, and uses React hooks for state management.
As LLMs continue to evolve, they are becoming smaller and smarter, enabling them to run directly on your phone. Take, for instance, the DeepSeek R1 Distil Qwen 2.5 with 1.5 billion parameters, this model really shows how advanced AI can now fit into the palm of your hand!
In this blog, we will guide you through creating a mobile app that allows you to chat with these powerful models locally. The complete code for this tutorial is available in our EdgeLLM repository . If you've ever felt overwhelmed by the complexity of open-source projects, fear not! Inspired by the Pocket Pal app, we will help you build a straightforward React Native application that downloads LLMs from the Hugging Face hub, ensuring everything remains private and runs on your device. We will utilize llama.rn , a binding for llama.cpp , to load GGUF files efficiently!
By the end of this guide, you will have a fully functional app that allows you to interact with your favorite models.
When selecting a model, consider both the parameter count and the quantization level. For example, a 7B model with Q2_K quantization might run better than a 2B model with Q8_0 quantization. So if you can fit a small model comfortably on your device try to use a bigger quantized model instead, it might have a better performance.
We could have implemented the api requests inside the handleDownloadModel function, but we will keep it in a separate file to keep the code clean and readable. handleDownloadModel calls the downloadModel function, located in src/api , which accepts three parameters: modelName , downloadUrl , and a progress callback function. This callback is triggered during the download process to update the progress. Before downloading we need to have the selectedModelFormat state set to the model format we want to download.
Let's start by defining the model formats our app is going to support and their repositories. Of course llama.rn is a binding for llama.cpp so we need to load GGUF files. To find GGUF repositories for the models we want to support, we can use the search bar on Hugging Face and search for GGUF files for a specific model, or use the script quantize_gguf.py provided here to quantize the model ourselves and upload the files to our hub repository.
Next, let's create a way to fetch and display available GGUF model files from the hub for our selected model format.
When a user selects a model format, we make an API call to Hugging Face using the repository path we mapped in our HF_TO_GGUF object. We're specifically looking for files that end with '.gguf' extension, which are our quantized model files.
Before we dive into building our app, let's talk about which models work well on mobile devices and what to consider when selecting them.
When downloading GGUF models, you'll encounter various quantization formats. Understanding these can help you choose the right balance between model size and performance:
React Native is a popular framework for building mobile applications using JavaScript and React. It allows developers to create apps that run on both Android and iOS platforms while sharing a significant amount of code, which speeds up the development process and reduces maintenance efforts.
Enabling LLM inference on edge devices reduces latency, preserves privacy, and lowers dependence on cloud infrastructure, which is critical as AI models grow in size and complexity.