Opening article
nanoVLM is a lightweight PyTorch toolkit that lets users train Vision Language Models (VLMs) on a free Colab notebook. Inspired by nanoGPT, it focuses on Visual Question Answering and uses a SigLIP vision encoder and a Llama‑3 style language backbone. The repository includes a simple training script, a Modality Projection module that aligns image and text embeddings, and a generate script for inference. A demo model was trained for about six hours on an H100 GPU using 1.7M samples from the cauldron dataset and published to the Hugging Face Hub.
nanoVLM is the simplest way to get started with training your very own Vision Language Model (VLM) using pure PyTorch. It is lightweight toolkit which allows you to launch a VLM training on a free tier colab notebook .
We were inspired by Andrej Karpathy ’s nanoGPT , and provide a similar project for the vision domain.
At its heart, nanoVLM is a toolkit that helps you build and train a model that can understand both images and text, and then generate text based on that. The beauty of nanoVLM lies in its simplicity . The entire codebase is intentionally kept minimal and readable , making it perfect for beginners or anyone who wants to peek under the hood of VLMs without getting overwhelmed.
We model nanoVLM after two well known and widely used architectures. Our vision backbone ( models/vision_transformer.py ) is the standard vision transformer, more specifically Google’s SigLIP vision encoder. Our language backbone follows the Llama 3 architecture.
Pixel shuffle reduces the number of image tokens, which helps reduce computational cost and speeds up training, especially for transformer-based language decoders which are sensitive to input length. The figure below demonstrates the concept.
While training, we use the following pre-trained backbone weights:
Now that we are familiar with the architecture, let's shift gears and talk about how to train your own Vision Language Model using train.py .
In this blog post, we cover the core ideas behind the project and provide a simple way to interact with the repository. We not only go into the details of the project but also encapsulate all of it so that you can quickly get started.
You can start training a Vision Language Model using our nanoVLM toolkit by following these steps:
Here is a Colab notebook that will help you launch a training run with no local setup required!
By simplifying VLM training and providing clear, minimal code, nanoVLM lowers the barrier for researchers and developers to experiment with multi‑modal models, accelerating innovation in AI applications such as image captioning and visual question answering.