The time axis: spacetime patches and the cost of duration

Treat a clip as a solid block. Width and height are two of its dimensions and time is the third, so the block gets longer the more seconds you ask for. An image model slices one thin sheet off the end and cuts that sheet into squares. A video model dices the whole block into small cubes, each covering a patch of frame across a short run of frames. OpenAI's Sora report calls these spacetime patches and presents them as the unit the model operates on, the video counterpart of the tokens a language model handles. Two consequences follow from the geometry. First, any cube can be compared with any other cube, including cubes far apart in time. That joint spatial and temporal attention is how a model keeps the end of a move consistent with its beginning. LTX-Video's authors argue that full spatiotemporal self-attention is what temporally consistent high-resolution video requires, and that they can afford it only because their autoencoder compresses so aggressively. Second, the number of cubes grows with frame area and with duration, and attention cost grows faster than linearly in that number. This is why video is priced differently from images. Movie Gen's largest video model is a 30 billion parameter transformer trained with a maximum context of 73,000 video tokens, which corresponds to 16 seconds at 16 frames per second. That figure, not artistic ambition, is what sets clip length: how many cubes fit in the context window and what attention across them costs. Not every design dices along time inside the diffusion model. Align Your Latents and Stable Video Diffusion instead add temporal layers to a pretrained image model and fine-tune them on encoded frame sequences. The compute pressure lands in the same place.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• a video model generates frames one after another the way a camera exposes them. Most current models denoise the entire clip at once, with all frames present in the same latent at every step.
• doubling the clip length doubles the cost. Token count grows with duration and attention cost grows faster than linearly in token count, so the bill grows worse than double.
• frame rate is effectively free because interpolation is cheap. In a spacetime-patch model, temporal resolution consumes the same token budget as spatial resolution. Some pipelines do add temporal super-resolution afterwards, but that is a separate model with its own failure modes.
• the model maintains an internal 3D scene and renders frames from it. It holds spacetime patches and attention between them. Any apparent geometry is a byproduct of appearance statistics, not a maintained representation.
• a longer clip is simply a longer version of the same generation. Length changes what fits in context, which changes how the model can hold the start and end of a move together.

Terms this introduces

spacetime patch

A token covering a small area of frame across a short span of frames, the basic unit a video model operates on.

sequence length

How many tokens one generation contains, growing with resolution and duration.

context length

The maximum number of tokens a model was trained to process at once, which caps clip length.

joint spatial-temporal attention

Attention computed across all positions in space and time at once, rather than separately per frame.

temporal layers

Blocks added to a pretrained image model so it can relate frames to each other, an alternative to dicing along time.