Why models fail: hands, text, physics and permanence
The failure modes that recur across every current video model, and the benchmarks built to measure them.
Imagine an exceptionally well-read forger who has copied millions of paintings of glass breaking and has never held a glass. The forgeries are convincing at a glance because the forger knows what the aftermath of breakage looks like. Ask for the moment of contact and the account falls apart, because there was never a glass, a hand or a floor, only pictures of them. Every characteristic failure of generative video is a version of this.
OpenAI's own Sora report says so directly: "Sora currently exhibits numerous limitations as a simulator. For example, it does not accurately model the physics of many basic interactions, like glass shattering. Other interactions, like eating food, do not always yield correct changes in object state." It also lists incoherencies developing in long samples and spontaneous appearances of objects. These are properties of the approach, not open bug tickets.
Benchmarks now measure the gap. Physics-IQ tests fluid dynamics, optics, solid mechanics, magnetism and thermodynamics, and its authors report that across current models physical understanding is severely limited and unrelated to visual realism. VideoPhy scores physical commonsense in everyday interactions across solid-solid, solid-fluid and fluid-fluid material pairs, using human evaluation. A separate study trained models on a deterministic 2D mechanics testbed and found perfect generalisation in-distribution, measurable scaling for combinatorial cases, and failure out-of-distribution, concluding the models generalise case-by-case, mimicking the nearest training example, and prioritise colour over size over velocity over shape when doing so. VBench breaks quality into sixteen dimensions, among them subject identity inconsistency, motion smoothness and temporal flickering.
The frame-level failures have the same root. Hands go wrong as wrong finger counts and impossible finger orientations. Small text and logos garble, which is why text-rendering work conditions on an explicit character layout. Weight, contact and object permanence read as approximately right and specifically wrong.
Write and board around the failure list rather than fighting it. Contact, impact, liquids being poured, hands manipulating small props, legible signage and anything that must persist across a cut are the shots that eat your generation budget. Move them off-model: a real plate, a practical insert, a 3D element, or a cut that hides the contact. Then use the failure list as your review checklist, because these errors are the ones a client notices on the second viewing and the ones that cannot be fixed with a note.
What people get wrong
Each of these is a belief worth dropping, followed by what is actually happening.
- these are bugs awaiting a patch. The Sora technical report presents physical-interaction and object-state errors as current limitations of the approach, alongside long-sample incoherence and objects appearing spontaneously.
- a more realistic-looking model has better physics. Physics-IQ reports that physical understanding across current models is severely limited and unrelated to visual realism, so photorealism is no guide to whether an interaction will be correct.
- scaling will fix physics. A study on a controlled mechanics testbed found in-distribution generalisation but out-of-distribution failure, and concluded that scaling alone is not sufficient for these models to recover fundamental physical laws.
- hands and text are the whole problem. Hands and small text are the visible tip; continuity across shots, weight, contact and object permanence fail more often and are harder to spot in a single viewing.
- one quality score tells you whether a model suits your shot. VBench exists because a single number hides the trade-offs; models differ dimension by dimension, so the useful question is how a model scores on the dimension your shot depends on.
Terms this introduces
- Object permanence
- Whether things persist correctly through occlusion and after leaving frame, which current models hold often but not reliably.
- Object state change
- The correct transformation of a thing under an action, such as food being eaten, which the Sora report lists as unreliable.
- Out-of-distribution generalisation
- Performance on cases unlike anything in training, where video models have been shown to fail rather than extrapolate a rule.
- Case-based generalisation
- Producing an answer by imitating the closest training example instead of applying an abstracted rule.
- Temporal flickering
- Frame-to-frame instability in texture, colour or detail, tracked as its own evaluation dimension in VBench.
Where this shows up in the directory
Models whose own documentation names this. It is evidence the vendor considers it relevant, not a judgement that the model is good or bad at it.
Whose job this changes
Craft roles whose page discusses this directly. Follow one to see what the person actually decides.