Training a coding model to paint watercolours with TRL and OpenEnv
Surya Narreddi recently shared a viral video demonstrating a language model painting watercolours by writing JavaScript code through the p5.brush library. A developer from Hugging Face attempted to reproduce this idea using TRL and OpenEnv, releasing all code, datasets, and trained models. The project explores whether reinforcement learning can be successfully applied to aesthetic preference rather than verifiable answers.
The Core Development
On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush , a library that "adds natural drawing tools to p5.js". The video went viral fast, over 1.5M views at the time of writing.
The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine . My attempt is on the engineering side, reproducing the recipe in the open with every piece published.
Note: for the context behind the project, told by Surya himself, watch this video of his thesis .
How It Works & Technical Architecture
Those are the weights Narreddi converged on. The pool defines taste here. That moves the work from tuning hyperparameters to building the set that decides what is beautiful.
Another change that cost me time to find was adjusting the LoRA parameters correctly. The usual target_modules list assumes a dense model, and Qwen/Qwen3.5-35B-A3B is a mixture of experts that names most of its projections differently, so the adapter was training ten layers out of forty. I solved this by changing it to all-linear , which reaches every linear layer. The routed experts in this architecture are fused tensors that even all-linear leaves frozen, but everything else gets an adapter, and that was enough to learn.
Strategic Perspective & Ecosystem Impact
In this article I try and reproduce his idea with TRL and OpenEnv . The reference pool dataset, the RL environment, the training scripts and the trained models, all open.
Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model , set two environment variables for the reward mix, and launch:
The rest of this article is the story of getting there, and every piece is in the repo .
Why it matters
This project explores the frontier of applying reinforcement learning to subjective aesthetic preferences rather than strictly verifiable tasks. By combining open-source tools like TRL and OpenEnv with human-rated image pools, it demonstrates how smaller models can be steered toward specific artistic styles.