How to fine-tune Llama 4 locally on a single GPU

Meta's Llama 4 is the most capable open-source language model to date, and thanks to QLoRA and other efficiency techniques, you can fine-tune it on a single consumer GPU. Here is how.

Hardware requirements

You will need at minimum an RTX 4090 with 24GB VRAM for the 8B parameter model, or an A6000 for the 70B model. Apple Silicon Macs with 64GB+ unified memory can also work using MLX.

The setup

Start with a clean Python 3.11 environment. Install PyTorch 2.3, transformers, peft, and bitsandbytes. The entire setup takes about 15 minutes.

Fine-tuning the 8B model on a domain-specific dataset of 10K examples takes approximately 4 hours on an RTX 4090. The results are impressively good for domain-specific tasks.