Diffusion models: generation by denoising

A diffusion model is a denoiser run far past its intended use. Start with the everyday version of that job: a noise-reduction pass on an underexposed night interior. The plugin looks at a grainy frame, estimates which part of the signal is grain, and subtracts it. A diffusion model is trained to make exactly that estimate, but on frames buried in far more noise than any sensor produces. Training is the straightforward half. Take a real image, add a known amount of noise, hand the result to the model, and ask it to say what the noise was. Because the amount added is known, there is an answer to grade against. Do that across enormous numbers of images at every noise level from a light dusting to pure static, and the model learns what grain looks like sitting on top of a photograph, which turns out to be the same as learning what photographs look like. This is the forward noising process, and it is fixed arithmetic chosen by the designers rather than anything the network learns. Generation runs the plugin in the direction it was never meant to go. Fill a frame with pure random noise, ask the model what the noise is, subtract most of the estimate, and look again. The residue is still mostly noise, but slightly less of it, and slightly structured. Ask again, subtract again, several dozen times. Structure accumulates because at each pass the most plausible thing to leave behind is the thing that resembles the training data. That is the learned reverse process. Note what the model never produces: the picture. Each pass it predicts noise, and the picture is what remains once the estimates are taken away.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• the model draws the image progressively, like a painter adding strokes to a blank canvas. In fact every step operates on the whole frame at once. What changes between steps is how much noise is left in the frame, not how much of the picture has been filled in.
• the noise is a random ingredient the model sprinkles in for variety. The noise is the starting material and the training signal. The model's only output is an estimate of the noise present, and the picture is the residue after those estimates are subtracted.
• the forward noising process is something the model learns. Only the reverse direction is learned. Adding noise is fixed arithmetic, which is what gives training a gradeable right answer at every noise level.
• diffusion models retrieve and blend training images. No image library is consulted at generation time. What is stored is a function mapping a noisy frame to a noise estimate, and the resemblance to training data comes from that function.
• more steps means a more detailed picture. Sampling converges. Past the point where each step moves the frame a negligible distance, extra steps buy render time and nothing else.

Terms this introduces

forward process

The fixed schedule of adding noise to a real image, used to manufacture training examples.

reverse process

The learned sequence of denoising steps that turns pure noise into a sample.

noise prediction

The model's actual output at each step: an estimate of the noise in the current input, not the finished image.

score

The direction in which a noisy image should move to become more like real data; the quantity score-based formulations estimate.

sample

One generation drawn from the model, which differs run to run because the starting noise differs.