Opening article
The Hugging Face blog describes a two‑phase training pipeline that turns the SmolVLM2‑2.2B‑Instruct vision‑language model into a GUI‑grounded agent. Phase 1 adds grounding by fine‑tuning on the smolagents/aguvis‑stage‑1 dataset, raising ScreenSpot‑v2 accuracy from 0% to 41%. Phase 2 adds agentic reasoning with smolagents/aguvis‑stage‑2, boosting accuracy to 61%. The authors also released unified action‑space datasets and a conversion tool, and replicated the method on a smaller nanoVLM‑460M model, achieving ~58% on ScreenSpot‑v2.
TL;DR: This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection here .
Graphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.
In this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.
Phase 1 leverages the smolagents/aguvis-stage-1 dataset, which introduces GUI grounding by pairing low-level instructions with diverse executable actions (expressed in code form). For example, a user/assistant turn in smolagents/aguvis-stage-1 follows the structure:
Each sample links a screenshot with multi-turn user/assistant interactions, enabling the model to learn fine-grained action grounding across dialogue turns. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.
Phase 2 uses the smolagents/aguvis-stage-2 dataset, which introduces agentic scenarios:
This section explains how we convert heterogeneous GUI actions format from multiple datasets into a single unified format . By standardizing function names, signatures, and parameters, we create consistent, high-quality data that forms the foundation for effective model training.
One of the primary challenges when working with multiple GUI automation datasets is the lack of standardization in action representations. Different datasets use varying function signatures, parameter naming conventions, and action taxonomies, making it difficult to train a unified model across diverse data sources.
You can use this tool to transform one action signature (function names, parameter names, and parameter value changes, ...) into another:
💡 We also reproduced the two-phase training on a much smaller VLM (nanoVLM-460M). Despite its reduced capacity, the model achieved ~58% on ScreenSpot-v2 , demonstrating that the training strategy scales down effectively, making it SOTA on ScreenSpot-v2 for this model size (460M parameters) . In addition, aguvis-stage-1 is already included in FineVision Dataset !
To maximize flexibility for different use cases, we developed the Action Space Converter ( utils/action_space_converter.py ), a tool that allows users to easily adapt from an action space to their own custom action vocabularies and naming conventions.
Here are real examples from our action conversion system ( preprocessing/action_conversion.py ) showing how we transform heterogeneous action representations into our unified format (grounding coordinates normalized to [0,1]):
To support the development of GUI agents, we’re open-sourcing everything: our complete pipeline, datasets, and trained model. You can reproduce our results, experiment with different models and architectures, or adapt our approach to new domains. The future of agentic AI depends on researchers like you pushing these boundaries further!
Rather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.
GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, …
Our approach leverages SmolVLM2-2.2B-Instruct as the baseline model, a small powerful vision-language model that initially has no grounding capabilities for GUI tasks. This makes it an ideal candidate to demonstrate the effectiveness of our training methodology. Through our two-phase training process, we first instill grounding capabilities in the model, then enhance it with agentic reasoning abilities using Supervised Fine-Tuning (SFT).
Demonstrating that lightweight VLMs can learn GUI grounding and reasoning with a reproducible pipeline shows that advanced UI automation can be achieved without large models, broadening practical AI deployment.