TL;DR: A diffusion model in AI is a type of generative model that learns to create images (and increasingly video, audio and even text) by reversing a noising process. During training, it takes real examples, gradually buries them in random noise, and learns to undo that damage one small step at a time. To generate something new, it starts from pure noise and “denoises” its way to a clean result, often guided by a text prompt. Diffusion is the technique behind most modern AI image generators.
If you’ve typed a prompt into an AI image tool and watched a picture appear from a blur, you’ve seen a diffusion model at work. So what is a diffusion model in AI, exactly, and why did it overtake older approaches? This guide explains the idea in plain English, with no maths required, plus where the technique is used and where it falls short.
What is a diffusion model? The simple version
Imagine taking a photo and adding a tiny bit of static. Then a bit more. Repeat that hundreds of times and the photo becomes pure TV snow, with nothing of the original left.
A diffusion model learns to run that film backwards. Shown a slightly noisy image, it learns to predict what noise was added so it can remove it. Chain enough of those small clean-up steps together and the model can start from random static and end with a brand-new, realistic image that never existed before.
The name comes from physics. The original 2015 work by Jascha Sohl-Dickstein and colleagues drew on ideas from nonequilibrium thermodynamics, where diffusion describes how things like ink spread out in water. The breakthrough paper for image quality came in 2020, when Jonathan Ho, Ajay Jain and Pieter Abbeel published Denoising Diffusion Probabilistic Models (often shortened to DDPM). They showed diffusion models could produce high-quality images, sometimes better than published results from other kinds of generative model.
How diffusion models work, step by step
1. The forward process: adding noise
During training, the model takes a real image and adds small amounts of Gaussian noise (random static) over many steps until nothing recognizable remains. This part isn’t learned; it’s a fixed recipe. In the DDPM paper, the researchers used 1,000 steps.
2. The reverse process: learning to denoise
The neural network is trained on a simple task: given a noisy image and the step number, predict the noise that was added. It practices on millions of examples at every noise level, from “barely fuzzy” to “pure static.” In the DDPM paper, the network doing this job was a U-Net, an architecture originally designed for image segmentation.
3. Generation: starting from pure noise
To make a new image, the model starts with random noise and repeatedly applies what it learned, removing a little predicted noise at each step. Interestingly, the DDPM authors observed that large-scale features appear first and fine details last, a bit like a sketch slowly being filled in.
4. Guidance: steering with a prompt
On its own, a diffusion model generates random examples of whatever it was trained on. To follow a text prompt, the model also receives a representation of your words, usually produced by a text encoder built on the transformer architecture. Those words are turned into embeddings, numerical representations of meaning, that nudge every denoising step toward “a watercolor of a fox” rather than anything else.
Latent diffusion: the trick that made it practical
Early diffusion models worked directly on pixels, which is expensive. The Latent Diffusion Models paper by Robin Rombach and colleagues (2021) noted that training the most powerful pixel-based models often took hundreds of GPU days, and generating 50,000 samples could take about five days on a single A100 GPU.
Their fix was to run diffusion in a compressed “latent space.” An autoencoder first squeezes the image into a much smaller representation, the diffusion model works there, and a decoder turns the result back into a full image. The paper also added cross-attention layers so the model could be conditioned on text, and trained a 1.45-billion-parameter text-to-image model on the LAION-400M dataset. This latent approach is the research foundation of Stable Diffusion, one of the best-known open image generators, which you can even run on your own PC.
Diffusion models vs GANs vs autoregressive models
| Diffusion models | GANs | Autoregressive models (e.g. LLMs) | |
|---|---|---|---|
| How they generate | Start with noise, refine over many steps | A generator makes an image in one pass while a discriminator judges it | Predict one token after another |
| Training | Stable, simple objective (predict the noise) | Can be unstable; two networks compete | Stable; predict the next token |
| Speed | Slower, because of many denoising steps | Fast, single pass | Depends on output length |
| Typical use | Images, video, audio, some new text models | Faces, style transfer, older image tools | Text, code, chat |
The Latent Diffusion paper notes that, as likelihood-based models, diffusion models don’t suffer from the mode collapse and training instability that GANs can show. The trade-off is speed: the authors say their sequential sampling process is still slower than GANs.
Where diffusion models are used
- Text-to-image generation: turning prompts into pictures, illustrations and concept art.
- Inpainting and editing: filling in or replacing part of an image, like removing an object.
- Super-resolution: upscaling low-resolution images, one of the tasks the Latent Diffusion paper tests.
- Video and audio: the same denoising idea extends to frames of video and waveforms of sound.
- Text: a newer wave of diffusion language models generates many tokens in parallel instead of one at a time. We covered one example in our piece on Inception’s Mercury 3 diffusion LLM.
Limitations of diffusion models
- Compute cost: many denoising steps mean more processing per image than single-pass models, though faster samplers such as DDIM can produce good results in far fewer steps than the original 1,000.
- Fine detail and precision: the Latent Diffusion authors note that compression can become a bottleneck for tasks that need fine-grained pixel accuracy.
- Bias and misuse: both papers flag that generative models reflect biases in their training data and make it easier to create convincing fake images.
- Training data questions: large image datasets scraped from the web raise ongoing copyright and consent debates.
Key terms to know
- Noise schedule: the recipe for how much noise is added at each step.
- Sampling steps: how many denoising passes the model makes when generating; fewer is faster but can lower quality.
- Latent space: the compressed representation where latent diffusion models do their work.
- Guidance scale: how strongly the output is pushed to match your prompt. Classifier-free guidance, introduced by Jonathan Ho and Tim Salimans, is a widely used method.
FAQ
Is ChatGPT a diffusion model?
ChatGPT’s text models are large language models that generate text token by token, which is autoregressive, not diffusion. Diffusion is the dominant approach for image generation.
Why is it called “diffusion”?
The idea borrows from physics, where diffusion describes particles spreading out until evenly mixed. Adding noise until an image is “fully mixed” mirrors that, and the model learns to reverse it.
Are diffusion models only for images?
No. Images are the best-known use, but the same technique is applied to video, audio and, more recently, text.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
