Transformers and DiT: why the backbone changed

The two backbones differ the way two crew structures differ. A U-Net runs a chain of command. Detail passes up through a hierarchy of resolutions, decisions get made at the coarsest level where the whole frame fits in view, then results pass back down. It works, and it carries a built-in preference for local neighbourhoods, because a convolution only sees nearby positions in any one layer. A transformer puts everyone in the same room and lets any participant address any other directly. The mechanism is attention, from Attention Is All You Need: each position issues a query about what it needs, every position advertises what it holds, and each position takes a weighted blend of everything available. No hierarchy, no locality preference. What counts as a participant is a token. In language a token is a piece of a word; in images the frame is diced into patches and each patch becomes a token. DiT made the swap explicit. Keep latent diffusion, remove the U-Net, and denoise latent patches with a transformer. The result that mattered was not one benchmark number but a trend: DiTs with more compute per forward pass, whether through greater depth, greater width or more input tokens, consistently reached lower FID. A trend like that is a plan, because it says spending more will pay off predictably. Video is where the plan paid best. A clip is a very long token sequence, and the thing a video model must get right is agreement between positions far apart in time, which is what all-to-all attention is for. Current open video models are diffusion transformers: CogVideoX with an expert transformer for text and video, HunyuanVideo at over 13 billion parameters, and Wan, whose authors report scaling behaviour in both data and model size.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• attention means the model looks where a human would look. Attention is weighted retrieval between positions inside the network. It sometimes correlates with salience, but it is not a gaze and not an explanation.
• transformers replaced diffusion. DiT keeps the diffusion training objective and replaces only the backbone that performs the denoising.
• a token in an image or video model is an object, a word or a meaningful unit. It is a fixed-size patch of latent, drawn on a grid with no regard for what happens to be inside it.
• the DiT result proves bigger is always better. It reports a scaling trend between compute per forward pass and FID on image benchmarks. That is a prediction about spending, not a guarantee about any particular shot.
• U-Nets are obsolete. They remain in wide use for images and in some video pipelines, and Lumiere's space-time U-Net is a recent design. The shift to transformers is about scaling behaviour, not about one architecture being wrong.

Terms this introduces

attention

An operation where each position pulls a weighted mixture of information from all other positions.

patch

A fixed-size square of latent treated as one token by the transformer.

DiT

Diffusion Transformer: a latent diffusion model whose denoising backbone is a transformer rather than a U-Net.

U-Net

The convolutional encoder-decoder backbone used by earlier diffusion models, which processes the frame through a hierarchy of resolutions.

scaling behaviour

The observed relationship between compute or model size and output quality, which lets teams predict the return on more spending.