Hugging Face has announced support for running GGUF models efficiently within the transformers library, allowing users to leverage checkpoints scaled for laptop memory through familiar APIs. Developed by the llama.cpp team, the GGUF format packages model weights, metadata, tokenizer information, and chat templates into a single file while supporting various quantization levels to trade precision for a smaller memory footprint. To bring performance close to llama.cpp, Hugging Face reuses the underlying ggml kernels through the kernels library and reduces overhead in generation, with an initial focus on local inference on Apple Silicon starting with the Qwen3.5 architecture. Users can load a GGUF model by passing its Hub model ID and filename as gguf_file to from_pretrained without extra configuration, as weights stay packed on Metal and automatically load compatible layer and attention kernels. The same checkpoints can also be used with transformers serve to expose an OpenAI-compatible API. While llama.cpp remains the recommended engine for dedicated, efficient local inference, this integration allows developers to use GGUF checkpoints directly inside transformers and provides a path to bring ggml performance and acceleration to custom variants, research models, and multimodal applications that lack dedicated llama.cpp implementations.
Transformers now runs llama.cpp quants
Hugging Face has added support for running GGUF models efficiently within the transformers library using familiar APIs. The initial focus centers on local inference on Apple Silicon, utilizing underlying llama.cpp ggml kernels and reduced generation overhead. This integration allows users to load checkpoints sized for laptop memory directly from the Hub.
Why it matters
Integrating ggml kernels into transformers bridges the gap between model definition and efficient local inference, potentially extending performance acceleration to new architectures, research models, and multimodal workflows without requiring full separate runtime implementations.