Memory Bandwidth (GB/s)
Memory bandwidth measures how much data your memory can feed to the processor each second, expressed in gigabytes per second (GB/s). Picture VRAM capacity as the size of a warehouse, and memory bandwidth as the width of the highway connecting that warehouse to the factory floor—no matter how fast the workers are, they can only assemble products as fast as trucks deliver the parts.
For local AI text generation, memory bandwidth is the single biggest driver of your tokens-per-second speed. Every time a model generates a single word, the GPU has to read the entire model's weights from memory, run a quick calculation, and move on to the next word. Because modern graphics cards crunch math far faster than memory can supply data, the GPU spends most of its time waiting on the memory highway—a bottleneck engineers call being memory-bandwidth bound.
You can roughly estimate how fast a rig will generate text by dividing its memory bandwidth by the file size of your model. An RTX 3090 with 936 GB/s of bandwidth reading a 20GB model file can theoretically cycle through those weights around 45 times per second—yielding roughly 35–40 tokens per second in real-world use. Step up to an RTX 5090 at 1,792 GB/s, and you nearly double that highway width to push 60+ tokens per second on the exact same model. By contrast, standard desktop DDR5 RAM tops out around 60 to 90 GB/s, which explains why spilling a model over into system RAM drops text generation to a crawl.
Looking for verified hardware rigs?
Check our real-time £/GB VRAM index to find systems optimized for local model inference.