LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) have made fine-tuning large language models accessible on consumer hardware. They are the standard approach for practitioners who don't have access to multi-GPU clusters.
Instead of updating a large weight matrix W directly, LoRA adds two small matrices alongside it — A (W_rows × r) and B (r × W_cols) — where r is a small rank. Only A and B are trained; W stays frozen.
After training, A × B can be merged directly into W. The final model behaves identically to a fully fine-tuned model at inference time — no extra compute, no larger model file beyond the adapter size.
A 4096×4096 weight matrix has 16M parameters. LoRA with r=16 parameterizes the same-shaped update using only 2 × 4096 × 16 = 131K parameters — about 120x fewer.
Fewer parameters, lower memory, faster training. Good for simple behavioral adaptation like output format or tone. Start here.
More capacity for complex task adaptation or large models. Higher memory and training cost. Diminishing returns above r=64 are well-documented.
QLoRA quantizes the frozen base model weights to 4-bit precision, reducing its memory footprint by ~4x, while keeping the LoRA adapters in 16-bit. The quality loss from 4-bit base quantization is minimal.
Fine-tune a 13B model on a single 24 GB consumer GPU (RTX 3090/4090). Fine-tune a 70B model on a single 80 GB A100. Without QLoRA, both would require multi-GPU infrastructure.
Hugging Face PEFT — The standard library for LoRA. Integrates with Transformers and TRL.
bitsandbytes — Handles 4-bit quantization for QLoRA.
Axolotl — Training framework that wraps PEFT/TRL with sensible defaults and YAML config. Recommended for most fine-tuning projects.
Ask the AI assistant about LoRA, QLoRA, rank selection, or the tooling ecosystem for fine-tuning.