Conditioning and control: everything you hand the model besides noise

A generative model has one input it cannot refuse: noise. Everything else is conditioning, information injected alongside the noise to bias what the denoiser converges on. Picture a scenic artist at a board with overlays taped over the surface: a written note in the margin, a photograph pinned at the top left, a sheet of tracing paper marking where the ground plane and the far wall sit, a stick figure showing where the arms go. The artist must satisfy every overlay at once, and the more overlays, the less room to invent. That stack is the whole control surface.

The written note is the text prompt, encoded and mixed in through cross-attention layers, the mechanism latent diffusion introduced for general conditioning inputs. The pinned photograph is image-to-video: a start frame the generation must begin from, which is how Stable Video Diffusion and most commercial image-to-video work. Pin a second photograph at the end and you have first and last keyframe conditioning, which turns generation into interpolation between two frames you chose. Mask a region and regenerate only inside it and that is inpainting; extend the board past its edge and that is outpainting. The tracing paper is structure conditioning: ControlNet adds a trainable copy of the encoder joined by zero-initialised convolution layers, so depth maps, Canny edges and human pose steer geometry without retraining the base model, with a conditioning scale that sets how hard each overlay pushes. VideoComposer carries the same idea across time by feeding motion vectors from compressed video as an explicit temporal condition. Video-to-video drives the entire stack from an existing clip.

Name the limit plainly. These overlays approximate a lens, a T-stop, a focus pull and a light on a stand. A depth map is a hint about where surfaces sit, not a 32mm at T2 with a marked focus pull.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• a good enough prompt can specify anything. In practice text is one conditioning channel competing with image, structure and motion inputs, and spatial precision comes from the image-side overlays, not from more adjectives.
• image-to-video means the model keeps your image intact. It uses the frame as a starting condition and re-synthesises everything after it, so wardrobe, set dressing and background text drift as the clip runs.
• ControlNet-style depth and pose conditioning give you camera and lighting control. They constrain where surfaces and limbs are; the lens character, exposure and light direction still come out of the model's learned appearance.
• stacking more controls always helps. Multiple conditionings can contradict each other, and the standard advice with multi-ControlNet setups is to mask the inputs so they do not overlap and to tune each conditioning weight separately.
• inpainting a face fixes it. Inpainting resamples inside the mask with the same priors that produced the error, so the usual fix is a better reference or a different seed rather than a repeated local repair.

Terms this introduces

Conditioning

Any input other than the initial noise that biases what the model generates, including text, images, masks, structure maps and camera trajectories.

Cross-attention

The layer type that lets a diffusion model read a conditioning signal such as an encoded text prompt while denoising.

Structure conditioning

Conditioning on a spatial map derived from a reference, such as depth, edges or human pose, to fix geometry while leaving appearance to the model.

Conditioning scale

A weight that sets how strongly a structure control overrides the model's own tendencies, traded against image quality.

Outpainting

Generating new content beyond the existing frame edges, used to extend a plate or change aspect ratio.