The AI tools for Art Newsletter - Issue 1

The AI space is moving so fast it’s hard to believe that a year ago we still struggled to generate people with the correct amount of fingers 😂.

The last couple of years have been pivotal for open source models and tools for artistic usage. AI tools for creative expression have never been more accessible, and we’re only scratching the surface. Join us as we look back at the key milestones, tools, and breakthroughs in AI & Arts from 2024, and forward for what’s to come in 2025 (spoiler 👀: we’re starting a new monthly roundup 👇).

What were the standout releases of creative AI tools in 2024? We'll highlight the major releases across creative and artistic fields, with a particular focus on open-source developments in popular tasks like image and video generation.

2024 was the year we shifted paradigms of diffusion models - from the traditional Unet based architecture to Diffusion Transformer (DiT), as well as an objective switch to flow matching.

Back to practice : First to announce the shift was Stability AI with Stable Diffusion 3 , however it was HunyuanDiT that became the first open source model with DiT architecture. This trend continued with the releases of AuraFlow , Flux.1 and Stable Diffusion 3.5 .

However, an upper bound to the quality of the fine-tuned models is naturally the base model from which it was fine-tuned. In that sense, we can’t neglect Stable Diffusion XL, which was also a significant marker in personalization for open source image generation models. A testimony to that is that even now, many of the popular techniques and models for personalization and controlled generation are based on SDXL. The advanced abilities of SDXL (and models that were released after with similar quality) together with the growing understanding of the semantic roles of different components in the diffusion model architecture raises the question - what can we achieve without further optimization?

cue in the rain of zero shot techniques - 2024 was definitely the year when generating high quality portraits from reference photos was made possible with as little as a single reference image & without any optimization . Training free techniques like IP adapter FaceID , InstantID , Photomaker and more came out and demonstrated competitive if not even superior abilities to those of fine-tuned models.

Based on a ComfyUI workflow by Nathan Shipley , Flux style shaping combines Flux [dev] Redux and Flux [dev] Depth for style transfer and optical illusion creation.

ComfyUI-to-Gradio - a step-by-step guide on how to convert a complex ComfyUI workflow to a simple Gradio application, and how to deploy this application on Hugging Face Spaces ZeroGPU serverless structure, which allows for it to be deployed and run for free in a serverless manner read more here .

YuE - series of open-source music foundation models for full song generation. YuE is possibly the best open source model for music generation (with an Apache 2.0 license!), achieving competitive results to closed source models like Suno.

Over 2 years since the OG stable diffusion was released and made waves in image generation with open source models, it’s now safe to say that when it comes to image generation from text, image editing and controlled image generation - open source models are giving closed source models a run for their money.

TD;LR - diffusion models and Gaussian flow matching are equivalent. Flow matching proposes a vector field parametrization of the network output that is different compared to the ones commonly used in diffusion models previously.

Among many pivotal moments in the (not so long) history of open source image generation models, it’s safe to say that the release of Flux.1 was one of them. Flux [dev] achieved a new state-of-the-art, surpassing popular closed source models like Midjourney v6.0, DALL·E 3 (HD) on various benchmarks.