Understanding GPU VRAM Allocation During LLM Fine-Tuning
Fine-tuning Large Language Models (LLMs) requires substantially more GPU video memory (VRAM) than simple inference. While running inference on a 7B model in 4-bit only consumes ~5 GB of VRAM, fine-tuning introduces several major memory consumers:
Key Memory Consumers in Deep Learning Training
- Model Weights: In FP16/BF16, each parameter takes 2 bytes (e.g., an 8B model takes ~16 GB). In 4-bit NormalFloat (QLoRA), the base model is compressed down to 0.5 bytes per parameter (~4 GB for 8B).
- Optimizer States: Standard 32-bit AdamW stores 2 states per trainable parameter (first momentum $m_t$ and second momentum $v_t$) in 32-bit float, taking 8 bytes per trainable parameter. Using 8-bit AdamW (bitsandbytes) reduces this to 2 bytes per parameter.
- Gradients: Backpropagation computes and stores gradients in 16-bit float (2 bytes per trainable parameter).
- Activations: Intermediate tensors generated during the forward pass and held until backpropagation. Activation memory scales linearly with sequence length ($S$), batch size ($B$), and number of layers ($L$). Enabling Gradient Checkpointing discards intermediate activations and recomputes them on the fly during the backward pass, reducing activation memory by ~60% to 75% at the cost of ~20% slower training throughput.
- CUDA Framework Buffer: PyTorch and CUDA runtime allocate internal workspace buffers and memory fragmentation buffers (~1.2 GB to 2.5 GB).
Training Strategy Comparison
- QLoRA (Quantized Low-Rank Adaptation): Freezes the base model in 4-bit NF4 and trains small low-rank adapter matrices in 16-bit. Allows fine-tuning an 8B model on a single consumer GPU with 12GB–16GB VRAM (e.g. RTX 3060 / 4070 / 4080).
- LoRA (Low-Rank Adaptation): Keeps the base model in 16-bit BF16/FP16 without quantization, training only low-rank adapter weights. Requires ~24GB VRAM (e.g. RTX 3090/4090 or A10G) for 8B models.
- Full Fine-Tuning: Updates 100% of the model weights, gradients, and optimizer states. For a 70B parameter model, full fine-tuning requires multi-GPU distributed clusters (e.g. 8x A100/H100 80GB with DeepSpeed ZeRO-3 or FSDP).