Latent space: working at proxy scale

Editors do not cut 8K camera masters. They cut proxies: small, cheap files that hold enough of the frame to judge performance, blocking and rhythm, then conform back to the masters at the end. Latent diffusion models work the same way and for the same reason. The 2022 latent diffusion paper moved denoising out of pixel space and into the compressed output of a pretrained autoencoder, on the grounds that pixel-space diffusion consumed hundreds of GPU days to train and stayed expensive to sample. A latent is that proxy. It is a grid much smaller than the frame, and each cell carries many channels of learned description rather than three colour values. Nothing in it is a pixel. A cell stands for a patch and encodes roughly what kind of surface, edge and texture belongs there. LTX-Video reports a video latent at a compression ratio of 1:192, with each token covering 32 by 32 pixels across 8 frames. What the compression buys is the whole pipeline. Attention cost grows faster than linearly with the number of positions, so shrinking the grid is what makes high resolution and multi-second video affordable at all. What it costs is exactly what proxies cost: the fine stuff. LTX-Video's authors say plainly that high compression limits the representation of fine detail. That is why small type, watch faces, embroidery, distant signage and background logos come back mushy or wrong. At proxy scale there was never a cell dedicated to the serif of a letter. Readers will also meet latents described as points in a space where similar things sit near each other. Treat that as a loose picture: low-dimensional plots of high-dimensional data distort distance and cluster size in well-documented ways.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• the latent is a small version of the picture, so a good upscaler could recover the missing detail. It is not an image at all, but a grid of learned descriptions. Detail that was never encoded can only be invented, not recovered.
• a larger output resolution means the model worked at that resolution. Denoising happens on the latent grid; pixel dimensions come from the decoder. Asking for more pixels does not add latent capacity.
• latent space is a map you can navigate by moving in a direction toward a concept. Some directions are meaningful, but the tidy pictures people show are low-dimensional projections, and projections of high-dimensional data mislead about distance and grouping.
• small text fails because the model cannot spell. Spelling is part of it, but compression is the harder limit. When a single token covers a 32 by 32 pixel patch, there is no room to carry the shape of a small glyph.
• all models share one latent space, so latents are portable. Each autoencoder defines its own; matching an existing latent space is a deliberate engineering goal, not a default.

Terms this introduces

latent

A compressed, learned representation of an image or clip that the diffusion model operates on instead of pixels.

latent diffusion

Running the denoising process inside a pretrained autoencoder's compressed space rather than in pixel space.

compression ratio

How many pixels of input each unit of latent stands for, trading detail for speed and length.

token

One position the model attends over; in image and video models, one patch of latent.

channels

The many values stored per latent cell, describing content rather than colour.