Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler
Hugging Face has published a beginner-focused guide on PyTorch profiling using torch.profiler. The post walks readers through analyzing execution traces, understanding CPU and GPU performance bottlenecks, and interpreting trace artifacts using tools like Perfetto.
The Core Development
Whether you are trying to squeeze more tokens per second out of a Large Language Model (LLM), shave milliseconds off inference, or just understand why your training loop runs slower than the spec sheet promises, the path eventually runs through profiling.
The catch is that profiling has a steep on-ramp. The traces are dense walls of colored rectangles. The events carry intimidating names. Most tutorials assume you can already read them. So even when we know we should be profiling, opening a trace can feel like a chore best left for later (or for someone else). This post, and the series it kicks off, is our attempt to lower that on-ramp.
We document the journey from a beginner's point of view. No prerequisites apart from basic PyTorch. Treat this as a leisurely read with some "Aha!" moments. The structure of the post is intentionally question-led: we open a trace, ask "wait, why is that happening?", and chase the answer until something clicks. By the end you should know:
How It Works & Technical Architecture
The matrix addition along with the matrix multiplication mimics how weights and biases interact in a neuron. This addition (pun intended) will help us understand how it paves the way for compilation later in the post .
Strategic Perspective & Ecosystem Impact
You don't usually have to write GPU kernels yourself; when you use a PyTorch operation, it is automatically translated to one or more kernels that do the job on GPU.
Here is the entire script that we use for the post: 01_matmul_add.py . We recommend opening this script in a separate tab and walk through the code step by step. We use the NVIDIA A100-SXM4-80GB GPU to run the scripts. It is really easy to setup a GPU on the Hugging Face infrastructure and experiment with the scripts using the Dev Mode with Spaces . One could also run the scripts with the Hugging Face Jobs pipeline . (Note: The traces in this blog were generated using PyTorch 2.13. If you are using a different version, some trace details like CompiledFxGraph might differ or be absent.)
As correctly quipped by Dr. Sara Hooker , just as we are primarily made up of water, Deep Neural Networks are primarily made up of matrix multiplies. As fundamental as they are, it would be a shame to start our profiling journey with anything else.
Why it matters
Understanding how to profile PyTorch code helps developers optimize Large Language Models and deep learning workflows by identifying bottlenecks, reducing overheads, and maximizing GPU utilization.