LoRAs and fine-tuning: teaching a model your subject

A trained model is a mixing desk with an enormous number of faders already set by someone else. Full fine-tuning means writing a new position for every fader, then storing and shipping the entire desk. Low-rank adaptation takes a different route: leave every original fader alone and add a small trim network in front of each channel strip, one that can only produce a limited family of coordinated nudges. It turns out most of the adjustment you want, teaching this face or this film stock, lives inside that limited family, so the trims are enough. That is a LoRA.

The mechanism is exactly this. LoRA freezes the pretrained weights and injects trainable rank decomposition matrices into each layer, and in the original language-model work that cut trainable parameters by four orders of magnitude against full fine-tuning of a 175-billion-parameter model, cut GPU memory roughly threefold, and added no inference latency, while matching or beating full fine-tuning on quality. For image and video models the practical consequence is a file of a few hundred megabytes that loads on top of a shared base, stacks with other adapters, and can be trained on one GPU.

What a dataset looks like is unglamorous and decides the result. A character LoRA wants a few dozen clean frames covering angles, expressions and lighting, with the background varied so the model does not learn the background. A style or look LoRA wants consistent grain, contrast and palette across varied subjects, so the model learns the treatment rather than the content. Captions matter: describe what varies, leave the thing you want bound to the trigger word undescribed. Overfitting announces itself. The subject appears when you did not ask for it, the training background creeps into every frame, prompt terms stop having any effect, and outputs converge on a handful of near-identical poses.

What people get wrong

Each of these is a belief worth dropping, followed by what is actually happening.

• a LoRA is a smaller model. It is a set of additional low-rank weights that only works on top of the specific base model it was trained against, so the base checkpoint is part of the deliverable.
• more training images always give a better LoRA. A small, clean, varied set beats a large set with repeated backgrounds or inconsistent lighting, because the model will happily learn the wallpaper along with the face.
• full fine-tuning is the quality ceiling and LoRA is the compromise. The original LoRA work reports performance on par with or better than full fine-tuning at a fraction of the trainable parameters, and adapters have the practical advantage of composing with other controls.
• if the LoRA works it will work at any weight. Adapter strength is a dial with a usable window; too low and identity slips, too high and the adapter overrides prompt, composition and camera behaviour.
• overfitting shows up as bad images. It usually shows up as loss of control: the trigger word stops being optional, the poses repeat, and the model can no longer place the subject anywhere it did not see during training.

Terms this introduces

LoRA

Low-rank adaptation: a training method that freezes the base weights and learns small rank-decomposition matrices inserted into each layer.

Rank

The size of the low-rank update, which sets how much the adapter can learn and how large the resulting file is.

Base checkpoint

The specific pretrained model an adapter was trained against and must be loaded with.

Trigger word

The token a fine-tune binds a subject or style to, used to call that concept up in a prompt.

Overfitting

Training so far into the dataset that the model reproduces its incidental details and loses responsiveness to prompts.