The latest update to the llama.cpp server introduces router mode, a feature that allows users to dynamically load, unload, and switch between multiple language models without restarting the server. This capability was added in response to popular requests for Ollama‑style model management and is built on a multi‑process architecture, ensuring that if one model crashes, the others remain operational. The server automatically discovers models stored in the llama.cpp cache (LLAMA_CACHE or ~/.cache/llama.cpp), including those previously downloaded via the llama-server -hf user/model command, and loads them into memory on the first request. Subsequent requests to the same model are served instantly because the model remains loaded. All loaded models operate with a 8192‑token context and full GPU offload, and users can define per‑model settings using presets. The built‑in web UI also supports model switching through a dropdown menu, making it easy to test different model versions or run multi‑tenant deployments without restarting the server. This feature aims to streamline development workflows and improve the flexibility of local LLM hosting.
New in llama.cpp: Model Management
The llama.cpp server now includes a router mode that allows dynamic loading, unloading, and switching between multiple models without restarting the server. This feature, inspired by Ollama, uses a multi‑process architecture so that a crash in one model does not affect others. Models are auto‑discovered from the llama.cpp cache and are loaded on first request, with subsequent requests served instantly. The built‑in web UI supports model switching via a dropdown, and all loaded models use a 8192‑token context with full GPU offload.
Why it matters
Dynamic model management simplifies A/B testing and multi‑tenant deployments, reducing downtime and resource overhead in local LLM hosting.