Text encoders and conditioning: how a prompt steers the picture

A prompt reaches the model as a board of reference cards rather than as a sentence. First the text is cut into tokens and each token becomes a vector produced by a text encoder, which is a separate network from the diffusion model. Two lineages matter. CLIP-style encoders are trained contrastively: given hundreds of millions of image and caption pairs scraped from the web, learn to predict which caption belongs to which image, which yields text vectors already aligned with picture content. Large text-only encoders such as T5 are trained on text alone. Imagen's authors found that scaling the frozen text encoder improved both sample fidelity and image-text alignment considerably more than scaling the diffusion model did, which is why current systems often carry both kinds. The SD3 architecture keeps separate weights for the text and image streams and lets information flow both ways, which its authors credit with better text comprehension and typography. Those vectors are the cards on the board. At every denoising step, every patch of the picture in progress looks over at the board and pulls what it needs. That look is cross-attention, introduced into diffusion models by the latent diffusion paper as the way one architecture can accept text, bounding boxes or other conditioning. Each patch weights the cards for itself: a patch becoming sky attends to different cards than a patch becoming a face. This is why phrasing has uneven effects. A word only does work if some patch chooses to attend to it. Concrete nouns with strong visual training signal pull hard. Negations, spatial relations and exact counts pull weakly, because there is no card reading not or three that a sky patch can act on. Reordering sometimes matters more than rewording.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• the model reads the prompt as a sentence and follows it as an instruction. It receives a set of token vectors, and each image patch weights those vectors for itself. There is no stage where the instruction is parsed and obeyed as a whole.
• longer, more detailed prompts always help. Adding tokens spreads attention across more of them and can weaken the ones already working. Specific concrete nouns usually beat length.
• negative words remove things from the image. Attention has no natural mechanism for not. Where systems do support negatives, they implement them as a separate conditioning pass, not as a word inside the prompt.
• any competent text encoder is interchangeable with any other. Encoders capture different things. Imagen's result that scaling the text encoder beat scaling the diffusion model shows how much rides on this, and models built on different encoders answer the same phrasing differently.
• the text encoder is trained alongside the image model. It is often frozen, trained separately on its own objective, and simply queried by the diffusion model.

Terms this introduces

text encoder

A separate network that converts prompt tokens into vectors the diffusion model can attend to.

contrastive training

Learning by matching images to their captions among many wrong options, as CLIP does.

cross-attention

The mechanism by which each image patch reads a weighted mixture of the prompt token vectors at every denoising step.

conditioning

Any extra input that biases generation, including text, a start image or structural maps.

frozen encoder

A text encoder whose weights stay fixed while the diffusion model trains around it.