Tokens Per Second Explained: How Fast Is Fast Enough?

Every generation request runs in two phases, and they have opposite performance profiles. The first is prefill, sometimes called prompt processing: the model reads your entire prompt at once and builds its internal state. Because all the prompt tokens are available up front, the GPU processes them in parallel, doing a large amount of matrix math in one pass. Prefill is compute-bound, meaning it is limited by how many math operations the chip can perform. On a modern GPU it can run at hundreds or even thousands of tokens per second, and it largely determines how long you wait before the first word appears.

The second phase is decode, the actual generation. Here the model produces one token at a time, and each new token depends on every token before it, so the work cannot be parallelized the same way. For each token the GPU must stream the model's weights and the growing key-value cache out of memory. Decode is memory-bandwidth-bound: it is limited not by math but by how fast data moves from VRAM to the compute cores, and GPU utilization during decode is often only twenty to forty percent. This is the number you watch scroll across the screen.

People conflate the two because both get reported as "tokens per second." A vendor quoting a huge throughput figure is often quoting prefill or aggregate batch throughput, not the steady single-stream decode rate a solo user experiences. When you compare hardware for interactive use, insist on the decode number.

There is a natural ceiling on useful single-stream speed, and it is set by how fast you read. Average adult reading runs around 200 to 300 words per minute. At roughly 250 words per minute and the English token-to-word ratio, that works out to about five to eight tokens per second. In other words, a model generating in the single digits is already keeping pace with a careful reader.

Once generation comfortably exceeds reading speed, the bottleneck you feel stops being decode rate and becomes time to first token, the delay before anything appears at all. Past roughly ten tokens per second, output streams faster than you can absorb it, so further speed does not make a single conversation feel meaningfully snappier; a slow first token does. This is why a build that decodes at fifteen tokens per second with an instant first token can feel more responsive than one that decodes at forty but stalls for two seconds before starting.

Decode speed falls as model size rises, and the reason follows directly from the memory-bandwidth bottleneck. To produce each token the GPU has to read every active weight out of memory. A larger model has more weights, so each token requires moving more bytes, and more bytes per token at a fixed bandwidth means fewer tokens per second. This is why an 8-billion-parameter model decodes far faster than a 70-billion-parameter one on the same card, often by a factor that roughly tracks the size difference.

A useful back-of-the-envelope estimate: divide your hardware's usable memory bandwidth by the model's size in bytes to approximate a ceiling on decode tokens per second. A model occupying 16 GB on a card with around 1,000 GB/s of bandwidth has a theoretical ceiling near 60 tok/s; real numbers land lower because of the key-value cache, overhead, and imperfect bandwidth utilization. The estimate is rough, but it explains the pattern: doubling the model roughly halves decode speed, and doubling bandwidth roughly doubles it.

Three levers move tok/s in predictable directions once you understand the bottleneck. Quantization shrinks each weight from, say, 16 bits to 4 bits. Because decode is bandwidth-bound, fewer bytes per weight means fewer bytes to stream per token, so a 4-bit model typically decodes several times faster than the same model at full precision while also fitting in less VRAM. The tradeoff is a usually small drop in output quality.

Context length works the other way. The key-value cache grows with every token in the conversation, and the model must read that cache for each new token. A long prompt or a long-running chat inflates the cache into gigabytes, adding to the bytes streamed per token and dragging decode speed down as the conversation goes on. Batching is the lever that helps servers but not solo users: processing many requests together raises total throughput because decode's idle bandwidth gets shared across requests, yet it does not speed up, and can slightly slow, any single stream. A benchmark of aggregate batch throughput tells you little about how fast your one reply will feel.

If you remember one thing when comparing GPUs for local inference, make it this: for the speed you feel, memory bandwidth predicts decode tokens per second better than any compute figure. Two cards with similar teraflops but different bandwidth will deliver noticeably different generation speeds, with the higher-bandwidth card winning. Compute still matters for prefill and for how quickly the first token arrives, but the steady stream of generated text is gated by how fast weights and cache move out of memory.

This reframes hardware shopping. Rather than chasing the largest model your VRAM can hold, target a model whose quantized size lets your card's bandwidth deliver at least ten to twenty tokens per second for interactive reading, with more headroom only if you plan to run agents or batch workloads. On the model pages, note each model's size and recommended quantization, then divide by a card's bandwidth from the GPU comparison to sanity-check the decode rate before you buy. A card that comfortably clears your reading speed on the model you actually want is a better purchase than a bigger card running a model too large to stream quickly.