Training AI to Paint with Code: RL Meets Aesthetic Judgment

Most image-generation models treat pixels as the terminal output. Prompt in, image out — and if you want to change something, you start over.

Surya Narreddi and his collaborators took a different bet: train an LLM to generate images by writing code. The code is the artifact. You can edit it. You can tweak the brushstroke algorithm without regenerating from scratch.

The result is a system using Qwen 3.5 35B with GRPO (the same RL algorithm behind DeepSeek's math models) to produce p5.brush watercolor paintings. The training loop is elegant in its simplicity:

flowchart LR
    A[Prompt: "peach hibiscus in watercolor"] --> B[Qwen 3.5 35B writes p5.js]
    B --> C[Puppeteer renders to PNG]
    C --> D[Judge: pairwise vs 2 reference paintings]
    D --> E[GRPO update]
    E --> B

The tricky part — and the real contribution here — is the reward function. Aesthetic quality isn't verifiable like a math answer. Too rigid and the model converges on a boring local optimum. Too loose and it drifts into incoherence.

The team built a reference pool of 581 hand-rated watercolors. Every training iteration samples two references and has a judge model pick the better painting. The judgment feeds GRPO. No absolute scores, no rubrics — just pairwise preference, which is surprisingly robust for creative work.

This matters because it's a blueprint: code-generating models as creative tools with editability built in, where RL handles the parts that feel like taste rather than truth.

What happens when you apply this to UI design? To data visualization? To architecture diagrams? The pattern generalizes.