The Cheapest Way to Run a 70B Model Locally

A 70-billion-parameter model is awkward precisely because it sits just past the edge of a single consumer GPU. At Q4_K_M, the quant most people consider the floor for trustworthy output, a 70B model needs roughly 40 to 45GB of memory once you include the weights plus a modest context window. The largest consumer cards top out at 24GB. So the moment you commit to 70B locally, you are committing to either two GPUs, a Mac with large unified memory, or accepting that part of the model lives in slow system RAM.

That last option is the trap. A single 24GB card can hold a Q3 quant (around 34GB on disk, less in practice) only if you push a chunk of the layers onto the CPU. The problem is bandwidth: a 3090's VRAM moves data at roughly 900 GB/s, while system DDR5 manages 40 to 50 GB/s. Every layer that lives in RAM drags the whole model down to that slower speed. This is why partial offload feels so much worse than the memory math suggests it should.

Here are the realistic paths in order of upfront cost. The ranking flips if you weigh usability instead: the cheapest option to buy is the most painful to use, and the value sweet spot sits one rung up.

Think of the single-card route as the way to find out whether you actually like running models locally, and the dual-card route as the way to keep doing it. The Mac is for people who value silence and a single appliance over raw tokens per dollar, and cloud is for everyone who does this rarely enough that owning hardware makes no sense.

Quantization shrinks a model by storing its weights at lower precision. Q4 is widely treated as the point where 70B models stay coherent and useful; you give up a little, but for most chat, coding, and summarization work the difference against the full model is hard to notice. Dropping to Q3 to squeeze onto a single 24GB card is where the bargain stops being free.

At Q3 and below, the degradation is real and shows up first on the tasks you most want a 70B for: multi-step reasoning, instruction-following on long prompts, and code that has to be exactly right. The model gets vaguer, repeats itself more, and makes small logic errors a Q4 version of the same model would not. There is also a counterintuitive performance gotcha - very low quants can run slower than Q4 because of dequantization overhead, so you can end up with worse quality and no speed win. The practical rule: a smaller model at Q4 often beats a 70B at Q3. If Q3 is the only way you can fit 70B, ask whether a 32B-class model at Q4 would serve you better.

Buying hardware only pays off if you use it enough to amortize the cost. Two used 3090s plus the supporting parts land somewhere around $2,000 to $2,800 all in. At roughly $1.20/hr for a cloud A100 that comfortably runs a 70B, that is over 1,600 hours of rented compute before you break even on the cards alone - and that ignores electricity, the resale risk on used GPUs, and the weekend you spend building the machine.

If you run a 70B in concentrated bursts - a research project, a few weeks of heavy experimentation, an occasional batch job - renting almost always wins, and you get newer, faster silicon than you could afford to buy. Buying wins when usage is steady and ongoing: a daily coding assistant, a private model you query constantly, or anything where you care about data never leaving your machine. To put real numbers against your own usage, work it out with the cloud-vs-buy calculator at /calculator/cloud-vs-buy before you spend anything.

Treat these as mid-2026 estimates, not quotes - used GPU prices drift, cloud rates change weekly, and your local power cost matters. They are meant to show the shape of the decision, not promise a price.