Why the Savings Are So Extreme
A full update on a 4096×4096 layer is 16.7 million parameters. The LoRA version at rank 8 is 8 × (4096 + 4096) = 65,536 — about 0.4%. The saving comes from replacing a product of dimensions with a sum of them.
The memory win is even larger than the parameter count suggests, because optimizers like Adam store two extra values per trainable parameter. Freezing 99.6% of the weights removes that overhead almost completely — which is what lets a 7B model be fine-tuned on a single consumer GPU.
Choosing the Rank
Rank is the one hyperparameter that matters, and it is a capacity dial:
- r = 1–4 — extremely cheap; enough for style or tone adjustments.
- r = 8–16 — the common default. Handles most instruction-tuning and domain adaptation.
- r = 64–128 — for teaching genuinely new capabilities, approaching full fine-tuning behaviour at a fraction of the cost.
Higher is not automatically better: past the task's intrinsic rank you are adding parameters that mostly learn noise. Note also that A is initialised randomly while B is initialised to zero, so BA = 0 at the start and training begins from exactly the pretrained model.
The Practical Superpower: Swappable Adapters
Because the base model is untouched, a LoRA adapter is a small standalone file — often just a few megabytes. One copy of a 70B model in memory can serve many specialisations by swapping adapters, and they can be merged into the weights before deployment (W' = W + BA) so inference costs nothing extra.
Combine LoRA with a 4-bit quantized base and you get QLoRA, the recipe that made fine-tuning very large models possible on a single GPU.
Training a small update instead of the whole model
Fine-tuning a 7-billion-parameter model conventionally means updating all 7 billion weights. With Adam, that requires the weights, their gradients, and two optimiser moments — roughly 4× the model in memory, so about 112GB in 32-bit. Beyond most hardware.
LoRA changes what is trained. The pretrained weights are frozen, and for each target matrix a small pair of matrices is trained alongside it:
h = W₀x + BAx where A is r×d and B is d×r
W₀ never changes. A and B are the learned update, and their product has the same shape as W₀ while containing far fewer parameters, because r — the rank — is small.
For a 4096×4096 weight matrix with r = 8:
| | Parameters |
|---|
| Full matrix | 16,777,216 |
| A (8×4096) + B (4096×8) | 65,536 |
| Ratio | 0.4% |
Across a whole model, LoRA typically trains 0.1–1% of the parameters — a few million instead of billions.
Why a low-rank update is enough
The empirical claim behind the method is that the update needed to adapt a pretrained model to a task has low intrinsic rank. The base model already knows the language; adaptation is a comparatively small, structured change.
That is an observation rather than a theorem, and it has held up well: LoRA fine-tunes typically match full fine-tuning on task metrics, sometimes within noise, occasionally better because there is less capacity to overfit.
Two details make it practical:
A is initialised randomly and B to zero, so BA = 0 at the start and the model begins exactly as the pretrained one. Training then moves it gradually rather than disrupting it on the first step.
The adapter can be merged. After training, W₀ + BA can be computed once and stored as a single matrix, so inference has no extra cost. Or the adapter can be kept separate and swapped — which is what allows one base model to serve many tasks.
The parameters that matter
| Parameter | Meaning | Typical |
|---|
r | Rank of the update | 8–64 |
alpha | Scaling applied to BA | Often 2×r |
target_modules | Which matrices get adapters | Attention projections, often the FFN too |
dropout | Dropout on the adapter | 0.05–0.1 |
Rank is the capacity dial. 8 is enough for style and format adaptation; 32–64 for teaching genuinely new behaviour or a specialised domain. Larger is not automatically better — it adds capacity to overfit a small dataset.
Which modules matters more than people expect. Adapting only the query and value projections was the original recipe; adapting all attention projections plus the feed-forward layers generally works better, at more parameters.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters() # ~0.5% of total
Training two thin matrices instead of one fat one
LoRA freezes the model and learns a low-rank correction beside it. The whole method is one factorisation, and the arithmetic explains both why it saves so much memory and what it cannot represent.
Things to try
- Start at r = 8 on a 4096×4096 layer and read the trainable fraction — well under 1%.
- Drag the rank to 128. Parameters climb steeply but remain a small slice of the full matrix. Even a large rank is cheap.
- Widen d and k while holding rank fixed. Full fine-tuning grows quadratically; LoRA grows only linearly. The bigger the model, the better LoRA looks.
- Watch the ΔW preview at r = 1, then raise the rank. At rank 1 every row is a scaled copy of a single pattern; higher ranks can express genuinely independent structure.
- Switch to 70B and compare the adapter file size to the full model. That gap is why model hubs host thousands of adapters instead of thousands of full checkpoints.
Worth remembering
LoRA does not make the model smaller — it makes the update smaller. By constraining the change to a low-rank subspace, it converts fine-tuning from an infrastructure project into something that runs on one GPU and ships as a few-megabyte file, with no inference penalty once merged.
QLoRA: fine-tuning on one consumer GPU
LoRA reduces the trainable parameters. The frozen base model still has to be held in memory, and 7 billion parameters in 16-bit is 14GB before anything else.
QLoRA quantises the frozen base to 4-bit — about 3.5GB — while training the LoRA adapters in higher precision. Gradients flow through the quantised weights without updating them.
That combination is what put fine-tuning a 7B model within reach of a single 24GB card, and a 70B model within reach of two.
| Approach | Memory for a 7B model |
|---|
| Full fine-tuning (Adam, fp32) | ~110GB |
| Full fine-tuning (mixed precision) | ~60GB |
| LoRA (fp16 base) | ~16GB |
| QLoRA (4-bit base) | ~6GB |
The quality cost of the quantisation is small — measurably present, and generally acceptable given what it enables.
Adapters as deployable artefacts
Because a LoRA adapter is a few megabytes rather than gigabytes, it changes how fine-tuned models are shipped.
One base, many adapters. Serve a single base model in memory and load a small adapter per customer, per task or per tone. Serving frameworks support switching adapters per request, so a single deployment handles many fine-tunes.
Cheap distribution. A 20MB adapter can be versioned in git and downloaded in seconds. A 14GB full fine-tune cannot.
Cheap experimentation. Training several adapters and comparing them is affordable, where several full fine-tunes are not.
The trade: an unmerged adapter adds a small amount of inference latency, since BAx is computed alongside W₀x. Merging removes it and gives up the ability to swap.
When LoRA is not the answer
You need new knowledge, not new behaviour. Fine-tuning of any kind is a poor way to add facts. Use retrieval.
The domain is very far from pretraining. Adapting an English model to a low-resource language or to a genuinely novel modality may need more capacity than a low-rank update provides — consider full fine-tuning or continued pretraining.
You have very little data. Below a few hundred examples, few-shot prompting is often better than any fine-tuning, and much cheaper to iterate.
Absolute maximum quality with unlimited compute. Full fine-tuning still edges ahead on some tasks.
Questions people ask
What rank should I use? 8–16 for style and format; 32–64 for substantial behaviour change. Tune it on your evaluation set.
What is alpha for? It scales the adapter's contribution by alpha/r. Setting it to twice the rank is a common convention that keeps the effective scale stable as rank changes.
Which modules should I target? All attention projections plus the feed-forward layers is a strong default. Query and value only is cheaper and usually slightly worse.
Can I combine several adapters? Yes — adapters can be summed or weighted, which sometimes composes their behaviours and sometimes interferes.
Does merging change quality? No, if precision matches. Merging into a quantised base can lose a little.
Is LoRA as good as full fine-tuning? On most task metrics, effectively yes. Occasionally better, because there is less capacity to overfit.
Recap in one screen
- Freeze the pretrained weights; train a small low-rank pair
BA added to selected matrices. - Typically 0.1–1% of parameters are trained, with quality close to full fine-tuning.
B starts at zero so training begins from the pretrained model exactly.- QLoRA quantises the frozen base to 4-bit, putting 7B fine-tuning on a single consumer GPU.
- Adapters are megabytes: one base model can serve many, and merging removes the inference overhead.