nanoVLM: The simplest repository to train your VLM in pure PyTorch

nanoVLM is the simplest way to get started with training your very own Vision Language Model (VLM) using pure PyTorch. It is lightweight toolkit which allows you to launch a VLM training on a free tier colab notebook .

We were inspired by Andrej Karpathy ’s nanoGPT , and provide a similar project for the vision domain.

At its heart, nanoVLM is a toolkit that helps you build and train a model that can understand both images and text, and then generate text based on that. The beauty of nanoVLM lies in its simplicity . The entire codebase is intentionally kept minimal and readable , making it perfect for beginners or anyone who wants to peek under the hood of VLMs without getting overwhelmed.

We model nanoVLM after two well known and widely used architectures. Our vision backbone ( models/vision_transformer.py ) is the standard vision transformer, more specifically Google’s SigLIP vision encoder. Our language backbone follows the Llama 3 architecture.

Pixel shuffle reduces the number of image tokens, which helps reduce computational cost and speeds up training, especially for transformer-based language decoders which are sensitive to input length. The figure below demonstrates the concept.

While training, we use the following pre-trained backbone weights:

Now that we are familiar with the architecture, let's shift gears and talk about how to train your own Vision Language Model using train.py .

In this blog post, we cover the core ideas behind the project and provide a simple way to interact with the repository. We not only go into the details of the project but also encapsulate all of it so that you can quickly get started.

You can start training a Vision Language Model using our nanoVLM toolkit by following these steps:

Here is a Colab notebook that will help you launch a training run with no local setup required!