Distillation, quantization, pruning, and LoRA: how to make a 175B model behave like it fits on a laptop.
Higher T spreads probability mass — more inter-class information for the student
The techniques that put foundation model quality into production-sized budgets.
Training GPT-3 cost approximately $4.6 million in compute at 2020 prices. A single inference call on a 175B parameter model requires about 350 GB of GPU memory to hold the weights in FP16. Deploying a frontier model naively means either renting a server with eight A100s per user session or waiting seconds per token.
The efficiency research field emerged to solve this. Its central question: can we achieve 90% of a large model’s quality at 10% of its cost? The answer, increasingly, is yes — through a combination of knowledge distillation, quantization, pruning, parameter-efficient fine-tuning, and inference-time optimizations.
The key insight behind knowledge distillation is that a large model’s output probabilities contain more information than one-hot hard labels.
When a teacher model assigns probabilities [cat: 0.7, lynx: 0.2, dog: 0.05, fish: 0.03, car: 0.02] to an image of a cat, those soft probabilities reveal something hard labels cannot: that cats and lynxes are more similar than cats and cars. A student model trained on these soft targets learns class relationships, not just the correct label.
The distillation loss combines:
L = (1−α)L_hard + αT²L_soft
The temperature T controls how much the soft targets spread. At T=1 the teacher’s softmax is used directly. At T=5 the distribution flattens, exposing more inter-class similarity information. At T=10 the distribution is nearly uniform — useful entropy but weaker signal about the most likely class.
DistilBERT compressed BERT by 40% while retaining 97% of its performance on GLUE benchmarks. Llama 3 8B, trained on more data, distils the understanding of Llama 3 70B into a model that runs on a MacBook.
Quantization reduces the numerical precision used to store model weights, reducing memory and often improving inference throughput.
Standard training uses FP32 (32-bit float) or BF16 (16-bit bfloat). Quantization compresses these to:
Two approaches dominate:
The practical recipe for deployment: GPTQ or AWQ for 4-bit weight quantization, with per-channel scaling. This makes a 70B model accessible on a consumer workstation.
Pruning removes parameters that contribute little to the model’s output. The two flavors differ in what gets removed:
The lottery ticket hypothesis (Frankle & Carlin, 2019): a randomly initialized dense network contains a sparse sub-network that, if trained in isolation from the start, would match the full network’s accuracy. This suggests most of the capacity in large models is never needed.
LoRA (Hu et al., 2021) makes fine-tuning feasible on consumer hardware by freezing the pre-trained weight matrix W⊂0; and learning only a low-rank update:
W = W⊂0; + BA where B ∈ ℜd×r, A ∈ ℜr×k, r ≪ min(d,k)
For a 4096×4096 weight matrix with r=8, LoRA trains 2×4096×8 = 65,536 parameters instead of 4096² = 16.7M — a 256× reduction in trainable parameters.
The hypothesis is that weight updates during fine-tuning lie in a low-dimensional subspace (low intrinsic rank). LoRA exploits this: the full matrix update is approximated by the product of two small matrices that together capture the task-relevant adaptation.
At inference time, BA is added to W⊂0; once and the merged matrix is used normally — no overhead versus the base model. Multiple LoRA adapters for different tasks can be hot-swapped by subtracting one BA and adding another.
Training a 70B model on a single GPU would take centuries. Modern training distributes work across hundreds or thousands of GPUs using three forms of parallelism:
The ZeRO optimizer (Zero Redundancy Optimizer) eliminates redundancy in data-parallel training by sharding optimizer states, gradients, and parameters across GPUs instead of replicating them. ZeRO-3 reduces per-GPU memory proportional to the number of GPUs, enabling 175B model training on commodity clusters.
Even after compression, the autoregressive generation loop is slow: each token requires a full forward pass through the model. Three techniques attack this:
The combination of these techniques has dramatically moved what is possible on commodity hardware:
| Model | Params | MMLU Score | Runs on |
|---|---|---|---|
| GPT-3 | 175B FP16 | 70.1 | 8× A100 80GB |
| Llama 3 70B (INT4) | 70B INT4 | 79.5 | 2× RTX 4090 |
| Mistral 7B | 7B BF16 | 64.2 | RTX 3090 (single) |
| Phi-3 Mini | 3.8B BF16 | 69.0 | RTX 3060 (single) |
The frontier is not GPT-4 versus GPT-2. It is a well-trained 7B model versus an early 175B model — and the 7B wins on several benchmarks while fitting on a laptop GPU.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is the key insight behind knowledge distillation?
What does LoRA do, and why is r kept small?
What is the difference between weight-only and activation quantization?
What is speculative decoding and why does it speed up inference?
Distillation: soft targets at temperature T transfer inter-class knowledge. Quantization (GPTQ): 4-bit weights, 8x memory reduction. LoRA: low-rank matrix update, 100-1000x fewer trainable params. Speculative decoding: draft + verify = 2-4x inference speedup.