What Hardware Do You Need to Fine-Tune an LLM?

Inference is memory-cheap because it is a one-way trip. You load the weights, run a forward pass, and read off the output. A rough rule is about 2GB of VRAM per billion parameters at 16-bit precision, so a 7B model fits in roughly 14GB and a 13B in roughly 26GB, before you account for the key-value cache.

Training keeps almost nothing for free. On top of the weights you have to store gradients, which mirror the weights in size, and optimizer state, which for the standard AdamW optimizer keeps two running averages per parameter. That alone roughly quadruples the per-parameter footprint of weights plus gradients. Then there are activations: the intermediate values from the forward pass have to be held in memory until the backward pass consumes them, and they scale with batch size and sequence length.

Add it up and a full fine-tune in 16-bit precision lands near 16GB per billion parameters, roughly eight times what inference needs for the same model. That multiplier is the single most important fact in fine-tuning hardware planning. Everything below is about shrinking it.

Full fine-tuning updates every weight in the model. It produces the most flexible result and is the heaviest by far, which is why it lives on multi-GPU servers rather than desktops. LoRA (Low-Rank Adaptation) freezes the original weights and trains small adapter matrices alongside them, so gradients and optimizer state only cover the tiny adapters instead of the whole model. QLoRA goes one step further: it loads the frozen base model in 4-bit quantization and trains LoRA adapters on top, cutting the largest cost, the static weights, to a quarter of its 16-bit size.

The practical consequence is that QLoRA puts a 7B or 13B fine-tune within reach of a single 24GB card such as an RTX 4090, with headroom for reasonable batch sizes. LoRA in 16-bit is heavier but still single-card territory for those sizes. Full fine-tuning of even a 7B model needs the kind of VRAM you only get from datacenter cards. The rules of thumb below are approximate and move with sequence length, batch size, and quantization details, but they are the right mental model for sizing a build.

Almost every fine-tuning tool you will reach for assumes CUDA, NVIDIA's compute platform. The libraries that make low-VRAM training possible, including bitsandbytes for 4-bit and 8-bit quantization, the fused attention kernels, and the optimizer implementations, are built and tested against CUDA first. AMD's ROCm has improved and Apple Silicon can run small experiments, but the moment you hit an obscure kernel or a quantization path that was never ported, you lose hours that an NVIDIA card would have saved.

This is not a statement about which silicon is better in theory. It is a statement about where the ecosystem actually works today. For training specifically, where you are stacking quantization, custom kernels, and fast-moving libraries, a recent NVIDIA card with plenty of VRAM is the path of least resistance. A 24GB card such as a used 3090 or a 4090 is the entry point most people should target, and dual-GPU 4090 builds extend that to larger LoRA jobs and bigger batches.

VRAM is the binding constraint, but it is not the only one. System RAM matters because datasets are loaded and preprocessed on the CPU, the base model is often staged in RAM before moving to the GPU, and some workflows offload optimizer state to system memory to fit larger jobs. Treat 32GB as a sensible floor and 64GB as comfortable for serious work; offloading in particular trades plentiful RAM for scarce VRAM, but at a real speed cost.

Gradient checkpointing is the other major lever. Instead of keeping every layer's activations in memory for the backward pass, it stores only a few checkpoints and recomputes the rest on the fly. That can cut activation memory substantially in exchange for roughly 20-30 percent more compute time, which is often the difference between a job fitting on your card and not running at all. Smaller batch sizes, shorter sequence lengths, and 8-bit optimizers are the other dials to reach for when you are a few gigabytes short.

Buying a 24GB card makes sense if you fine-tune regularly, value local iteration, and want your data to stay on your own machine. But there is a clear line past which renting wins. Full fine-tuning of anything from 7B upward, multi-billion-parameter LoRA runs that exceed your VRAM, or a one-off job you will not repeat are all better served by an hourly cloud GPU than by hardware you have to house, power, and cool.

An H100 typically rents for roughly 2.50 to 4 dollars per hour on dedicated GPU clouds, with hyperscalers charging more and marketplaces sometimes less. At those rates an occasional large fine-tune costs tens of dollars rather than the thousands a comparable card would cost to buy. The break-even rule is simple: rent until your sustained utilization would keep an owned card busy a large fraction of the day over its useful life. Below that threshold, renting is cheaper and spares you the depreciation, power, and maintenance. A practical pattern is to develop and debug your pipeline on a local 24GB card with a small model and short runs, then rent an H100 only for the final full-scale job. The site's fine-tune workstation build is designed around exactly this local-iteration role, and the dual-4090 workstation build extends it to heavier LoRA work before you ever need to rent.