How Much VRAM Do You Need to Run Llama 70B?

The rule of thumb is simple. VRAM needed is approximately the number of parameters times the bytes per weight, plus the KV cache, plus some overhead. For a 70B model the weights dominate. At FP16 each weight is two bytes, so 70 billion parameters times two bytes is about 140GB just for the weights. Drop to 8-bit and each weight is one byte, halving that to roughly 70GB. Drop to a common 4-bit quant and you are near half a byte per weight, landing around 35 to 40GB.

On top of the weights you add the KV cache, which holds the attention keys and values for every token in your context, and a few gigabytes of overhead for activations, the CUDA context, and the inference runtime. For a 70B model at a typical context length the KV cache and overhead together add a handful of gigabytes. That is why the headline 4-bit number rounds up from the raw 35GB of weights to a planning figure closer to 40GB once you account for context and runtime slack.

Quantization is the single biggest lever you have. It trades a small, usually acceptable, loss in output quality for a large reduction in memory. The community has largely settled on 4-bit medium quants as the sweet spot for 70B models: they cut memory by roughly four times versus FP16 while keeping the model coherent and useful. Below that you keep saving memory, but quality degradation becomes more noticeable, especially on reasoning and code.

Here is a rough breakdown of where a 70B model lands by quantization, counting weights plus a little context headroom.

Different setups map onto those tiers cleanly. A single 24GB consumer card cannot hold a 4-bit 70B model with real context, so it only works at very aggressive 3-bit or lower quants with a tiny context window, and even then it is tight. A 48GB card such as an RTX A6000, or two 24GB cards like dual RTX 4090s pooled together, comfortably runs the 4-bit tier with room for context. An 80GB H100 handles the 8-bit tier on a single card. FP16 needs two 80GB cards or more. Apple Silicon is a special case: unified memory means the GPU can address most of system RAM, so a Mac with 64GB or 128GB of unified memory can load larger quants than its raw price suggests, though memory bandwidth caps how fast it runs.

Context length is the variable people forget, and it is the reason your VRAM use creeps up during a long session. The KV cache stores a key and value vector for every token currently in the context, for every layer. As your prompt and generated output grow, that cache grows linearly with the number of tokens. A short prompt costs almost nothing; a long document or a multi-turn conversation can add many gigabytes.

Llama 70B helps here by using grouped-query attention, which shares key and value heads across groups of query heads and cuts the KV cache by roughly eight times compared to older full multi-head designs. Even so, the growth is real. At a moderate context of around 32K tokens the 4-bit KV cache is on the order of eight to ten gigabytes, and pushing toward 128K can add tens of gigabytes on top of the weights. If you plan to use long context, budget for it explicitly rather than assuming the weight figure is the whole story. You can also quantize the KV cache itself to claw back some of that memory.

When a model does not fit entirely in VRAM, most inference engines will offload the overflow layers to system RAM and run them on the CPU. This lets you load a model that would otherwise be impossible, but the speed cost is severe. GPU memory bandwidth is an order of magnitude higher than system RAM bandwidth, and the CPU is far slower at the matrix math, so every offloaded layer becomes a bottleneck the whole generation has to wait on.

In practice, a 70B model running mostly on the GPU with only a few offloaded layers might still feel interactive, but a 70B model with most layers offloaded to RAM typically crawls at one to three tokens per second. That is slow enough that it is fine for a batch job you walk away from and frustrating for anything interactive. The lesson is that offloading is a fallback for fitting the model at all, not a strategy for good performance. If speed matters, size your VRAM so the model fits.

Speed expectations vary widely by setup. A single 48GB card running a 4-bit 70B model with the whole model resident in VRAM commonly lands in the rough range of fifteen to thirty tokens per second, depending on context length and the engine. Dual 4090s pooling 48GB run the same 4-bit tier and scale to roughly 85 to 90 percent of ideal because of the communication overhead between cards, so they perform similarly to a single big card while costing less.

An 80GB H100 running an 8-bit version will be faster still and is the comfortable choice when output quality at higher precision matters. A single 24GB card forced into heavy offloading sits at the bottom, in the low single digits. Treat all of these as ballpark figures; the only way to know your real number is to measure your own stack with your own quant, context length, and inference engine.