Full fine-tuning updates every parameter in the model. Parameter-Efficient Fine-Tuning (PEFT) freezes most parameters and updates only a small fraction — reducing memory and compute requirements dramatically while often matching full fine-tuning quality.
Every weight is updated at every training step. For a 7B parameter model, training state (weights + gradients + optimizer states) requires 60–80 GB of GPU memory — well beyond a single consumer GPU. Full fine-tuning of a 70B model requires a cluster of data center GPUs.
When you have multi-GPU infrastructure, need maximum adaptation capacity, and cost is not the primary constraint. Produces the best results but is impractical for most teams without significant GPU resources.
PEFT freezes most model weights and trains only a small set of additional parameters. The key insight: task adaptation requires only low-rank weight changes — a small fraction of the total parameter space. PEFT captures these changes efficiently.
Runs on consumer GPUs. Enables fine-tuning of large models (13B+) that wouldn't fit for full fine-tuning. Reduces catastrophic forgetting. Multiple task-specific adapters can be trained on the same base.
The most popular method — LoRA — adds small trainable matrices alongside the frozen weight matrices. Only these small matrices are updated. After training, they can be merged into the base weights with zero inference overhead.
Full fine-tuning — Multi-GPU cluster available, need maximum adaptation, fine-tuning a small model (<7B) that fits in memory. Best results, highest cost.
LoRA / QLoRA — Consumer or mid-range GPUs, fine-tuning models 7B and above, fast experimentation, or multiple adapters on one base. The practical default for most practitioners.
Ask the AI assistant about full fine-tuning vs PEFT, memory requirements, or how to choose the right approach for your hardware.