Home/Guides/How Much VRAM Do You Actually Need? (The Practical Sizing Guide)
Hardware 101••5 min read

How Much VRAM Do You Actually Need? (The Practical Sizing Guide)

Why does a 70B model crash on a 32GB graphics card? Learn the simple math behind model weights, quantization (Q4 vs Q8), and context window memory requirements.

When newcomers start shopping for an AI PC or workstation, the most common question is: "How many gigabytes of VRAM do I actually need?"

The short answer: It depends entirely on the size of the models you intend to run and the length of your conversation context. This guide breaks down the numbers with zero jargon.

The Basic Math Behind AI Model Sizing

An AI model consists of billions of mathematical numbers called weights or parameters.

Originally, each parameter is stored as a 16-bit number (2 bytes). That means an uncompressed 8-billion parameter (8B) model would need 16GB of memory just to load into RAM!

To make models manageable on consumer hardware, the community uses Quantization.

Quantization Explained (Q4, Q8, FP16)

Quantization compresses model weights by rounding numbers to fewer bits, reducing memory footprint by 50% to 75% with almost unnoticeable loss in accuracy:

  • 4-bit Quantization (Q4_K_M): Uses ~0.65 bytes per parameter. This is the industry standard sweet spot between quality and memory efficiency.
  • 8-bit Quantization (Q8_0): Uses ~1.1 bytes per parameter. Near-lossless precision, but requires nearly double the memory of Q4.
  • 16-bit Full Precision (FP16): Uses 2.0 bytes per parameter. Usually unnecessary for daily inference.

Don't Forget the Context Window (KV Cache)!

Loading the model is only part of the equation. As your conversation grows, the model must store past tokens in what is called the KV Cache (Key-Value cache).

  • A short 4,000-token conversation adds 1GB to 2GB of memory overhead.
  • A long 32,000 to 128,000-token context (e.g. ingesting an entire codebase or PDF document) can require an additional 4GB to 16GB of VRAM!

Always aim for at least 2GB to 4GB of headroom above the raw model size.

Quick Reference Sizing Cheat Sheet

Here is how much VRAM you need based on what you want to achieve:

1. Small & Fast Models (8B Parameters)

  • Popular Models: Llama 3.1 8B, Mistral 7B, Gemma 2 9B.
  • Minimum VRAM: 8GB – 12GB (at Q4).
  • Recommended VRAM: 16GB (leaves ample room for 32k+ context window).
  • Recommended Hardware: NVIDIA RTX 4060 Ti 16GB, RTX 5070, Apple Mac Mini 16GB/24GB.

2. Medium Intelligence & Coding Specialists (14B – 32B Parameters)

  • Popular Models: Qwen 2.5 32B, DeepSeek Coder 33B, Yi 34B.
  • Minimum VRAM: 20GB – 24GB (at Q4).
  • Recommended VRAM: 24GB – 32GB (accommodates 64k+ context and Q5/Q8 quantizations).
  • Recommended Hardware: NVIDIA RTX 3090 24GB, RTX 4090 24GB, RTX 5090 32GB, Apple Mac Studio (36GB/64GB).

3. Frontier Intelligence & Complex Logic (70B+ Parameters)

  • Popular Models: Llama 3.3 70B, DeepSeek R1 70B distilled, Qwen 72B.
  • Minimum VRAM: 44GB – 48GB (at Q4).
  • Recommended VRAM: 64GB – 128GB+ (allows running Q8 weights and large document context).
  • Recommended Hardware: Dual RTX 3090/4090 setup, Apple Mac Studio M2/M4/M5 Ultra (128GB - 256GB), Minisforum Ryzen AI Max+ (128GB).

What Happens If You Don't Have Enough VRAM?

If you try to load a 40GB model onto a 24GB graphics card, one of two things happens:

  1. Out of Memory (OOM) Crash: Many tools will simply refuse to run and throw an error.
  2. System Memory Spillover (Offloading): Modern tools like llama.cpp will place 24GB into your GPU and offload the remaining 16GB onto your computer's regular DDR5 RAM. While the model will run, every word output must wait on the slow system bus, dropping generation speed from 30 tokens/second to 1-2 tokens/second.

Summary Recommendation

  • If your budget is under £1,500: Aim for a system with at least 16GB VRAM to comfortably run all 8B models.
  • If your budget is £2,000 – £3,500: Look for 24GB to 32GB VRAM (RTX 4090/5090 or Mac Studio) to run top-tier 32B models.
  • If you need 70B frontier power: Prioritize unified memory workstations (64GB to 128GB+) rather than cramming multiple hot GPUs into a consumer PC case.

Explore our £/GB VRAM Index to find the best priced systems for each memory tier.

Rent Cloud GPU Power

Not Ready to Buy a Full Rig?

Don't want to spend £3,500+ upfront? Rent the GPU power instead.

Buying a dedicated AI workstation is a serious investment—and if you aren't ready to spend that much right now, you don't have to. Renting cloud GPUs gives you instant access to high-VRAM hardware like RTX 4090s and H100s on demand. Spin up an instance for fine-tuning or running massive 70B+ models, pay strictly for the minutes you use, and avoid the upfront hardware cost, electricity bills, and maintenance.

Vast.ai Global GPU Marketplace

Lowest $/hr
Unbeatable $/hr rates on community-hosted RTX 4090 & 3090 rigs
Deploy Pod →

RunPod Community & Secure Cloud

Community & Secure Cloud
RTX 4090 from $0.34/hr • H100 SXM from $1.99/hr (Billed in USD)
Deploy Pod →

FindBestRig independently benchmarks cloud GPU providers to compare rental economics against physical hardware.