Opening article
Hugging Face has added vision support to its smolagents framework, enabling vision‑language models (VLMs) to be used natively in agentic pipelines. The update allows agents to receive images either as a list in the run method or dynamically via callbacks that add screenshots to step logs. This opens possibilities such as autonomous web‑browsing agents that can interpret visual content on pages. The release includes examples using the Qwen2VL model and helium for browser automation, demonstrating how to integrate visual inputs into agent actions.
We have added vision support to smolagents, which unlocks the use of vision language models in agentic pipelines natively.
In the agentic world, many capabilities are hidden behind a vision wall. A common example is web browsing: web pages feature rich visual content that you never fully recover by simply extracting their text, be it the relative position of objects, messages transmitted through color, specific icons… In this case, vision is a real superpower for agents. So we just added this capability to our smolagents !
Teaser of what this gives: an agentic browser that navigates the web in complete autonomy!
As you can see, for use cases where images are dynamically retrieved (e.g. web browser agent), we support adding images to the model’s ActionStep , in attribute step_log.observation_images .
Let's make a model. We've added support for images in all models. Just one precision: when using TransformersModel with a VLM, for it to work properly you need to pass flatten_messages_as_text as False upon initialization, like:
🤔 How do we want to pass images to agents? Passing an image can be done in two ways:
For the case where we want to pass images at once, we added the possibility to pass a list of images to the agent in the run method: agent.run("Describe these images:", images=[image_1, image_2]) .
These image inputs are then stored in the task_images attribute of TaskStep along with the prompt of the task that you'd like to accomplish.
Integrating vision into agentic frameworks expands the range of tasks agents can perform, especially those requiring visual understanding such as web navigation and document analysis. This enhances the practical applicability of AI agents in real‑world scenarios.