Why does my 70B model load into VRAM but crash with Out of Memory (OOM) on long documents?

Model weights occupy a static footprint (e.g., 37.9 GB for LLaMA 3.3 70B at Q4). However, the Key-Value (KV) cache grows linearly with every token processed. At 128k context with FP16 KV-cache, the conversation memory alone requires 40.0 GiB (42.9 GB decimal)—larger than the model weights themselves! Total VRAM demand reaches ~81 GB, instantly exceeding a dual RTX 3090 (48GB) setup.

How does DeepSeek V3 achieve only 8.6 GB KV cache at 128k context?

DeepSeek V3 utilizes Multi-Head Latent Attention (MLA). Instead of caching separate key and value heads for each layer, MLA projects keys and values into a shared compressed low-rank latent vector (dc = 512) and a decoupled RoPE key vector (dr = 64). Total elements per layer per token is only 576 (compared to 2,048 for standard GQA), reducing 128k KV cache size by over 4.6x.

How can I fix a 128k context OOM without buying more hardware?

First, enable 8-bit quantized KV cache. In llama.cpp, add '-fa on -ctk q8_0 -ctv q8_0' (or in Ollama: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0). In vLLM, add '--kv-cache-dtype fp8'. This immediately cuts KV cache VRAM by 50% with negligible perplexity loss. Second, reduce maximum context to 32k tokens, which fits comfortably on 48GB hardware.

🧮 Mathematical Memory Derivation & Capacity Planner

128k Context VRAM Calculator Why Long Context Crashes Your GPU & How to Plan Concurrency

You loaded Meta LLaMA 3.3 70B onto your dual RTX 3090 (48GB) workstation. It initialized cleanly, took 38 GB VRAM, and answered short test prompts at 20 tok/s. Then you uploaded a 100-page PDF at 128k context, and your system instantly crashed with CUDA out of memory. Here is why the surprise explosion happens and the mathematical formulas to fix it.

💥 The Surprise Out-of-Memory Visual

LLaMA 3.3 70B @ 128k Context (FP16 KV)

⚠️

The Surprise Reality: At 128k context, conversation memory (40.0 GB KV Cache) is larger than the model itself (37.9 GB Weights)! Total memory is ~81 GB—leaving you 33 GB over the physical 48 GB limit of dual 3090s.

Weights 37.9GB (46%)

KV (Fits)

💥 33.0 GB OVERFLOW (OOM)

0 GB ↑ Physical Limit: 48 GB (Dual RTX 3090) Total Demand: ~81.4 GB

Model Weights 37.9 GB

KV Cache (128k FP16) 40.0 GB

Activations & Driver 3.5 GB

Hardware Deficit -33.4 GB

The Official KV-Cache Formulas (GQA vs MLA)

1. Grouped-Query Attention (GQA) — LLaMA 3.3, Qwen 2.5, Mistral

Standard LLMs

Every layer caches both keys and values for its grouped attention heads:

KV_Bytes = 2 × Layers × KV_Heads × Head_Dim × Bytes_Per_Elem × Tokens × Streams × Fill_Ratio

For LLaMA 3.3 70B (80 layers, 8 KV heads, 128 head dim): Elements per token = $2 \times 80 \times 8 \times 128 = 163,840$ elements. At 128k tokens ($131,072$) in FP16 (2 bytes): $163,840 \times 131,072 \times 2 = 42,949,672,960\text{ bytes} = \mathbf{40.0\text{ GiB}}$ ($42.9\text{ GB}$ decimal).

2. Multi-Head Latent Attention (MLA) — DeepSeek V2.5 / V3

4.6x Compression

MLA compresses keys and values into a single low-rank latent vector plus decoupled RoPE key vector. There is NO factor of 2!

KV_Bytes = Layers × (d_c + d_rope) × Bytes_Per_Elem × Tokens × Streams × Fill_Ratio

For DeepSeek V3 (61 layers, $d_c = 512, d_{rope} = 64$): Elements per token = $61 \times 576 = 35,136$ elements. At 128k tokens in FP16: $35,136 \times 131,072 \times 2 = 9,210,691,584\text{ bytes} = \mathbf{8.58\text{ GiB}}$. While KV cache is tiny, its 671B parameters require 380GB+ weights—mandating multi-GPU cloud nodes.

Exact 128k KV Cache Footprint by Model Architecture (1 Stream)

Model Architecture	Layers / Heads / Dim	Per-Token Elements	128k FP16	128k FP8 (vLLM)	128k Q8_0 (llama.cpp)	128k Q4_0 (llama.cpp)
Meta LLaMA 3.3 70B	80 / 8 / 128	163,840	40.0 GB	20.0 GB	21.25 GB	11.25 GB
Alibaba Qwen 2.5 72B*	80 / 8 / 128	163,840	40.0 GB	20.0 GB	21.25 GB	11.25 GB
DeepSeek V3 (671B MoE)	61 / MLA (512+64)	35,136	8.58 GB	4.29 GB	4.56 GB	N/A (MLA)
Meta LLaMA 3.1 8B	32 / 8 / 128	65,536	16.0 GB	8.0 GB	8.50 GB	4.50 GB

*Note: Qwen 2.5 72B is natively trained on 32k context; running at 128k requires YaRN RoPE context extension.

The 3-Tier Honesty Funnel: Free Fix, Rent, or Buy

Tier 1: 100% Free Software Fix 0.00 USD

Enable Quantized KV Cache & Right-Size Context

On your existing dual RTX 3090 (48GB) or single 24GB card, you can stop OOM crashes immediately by switching KV cache precision to Q8_0 or FP8, and clamping context to 32k tokens:

llama.cpp / llama-server command: llama-server -m llama-3.3-70b-instruct-q4_k_m.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0

Ollama environment variables: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama run llama3.3:70b

vLLM server flag: python -m vllm.entrypoints.openai.api_server --model ... --max-model-len 32768 --kv-cache-dtype fp8

Result: At 32k context with Q8_0 KV cache, LLaMA 3.3 70B total VRAM is ~45.7 GB, which fits 100% cleanly inside dual RTX 3090s with zero OOM crashes!

Tier 2: On-Demand Cloud Rental ~$0.79 – $1.99 / hr

Need Full 128k Context or Concurrency? Rent an 80GB GPU

If your workload genuinely requires the full 128,000 token context window or multiple concurrent user streams, renting an 80GB NVIDIA A100 or H100 by the hour is dramatically cheaper than spending $5,000–$15,000 on enterprise hardware:

Deploy on RunPod NVIDIA A100/H100 80GB from $0.79/hr

→

Rent on Vast.ai Community A100 80GB from $0.35/hr

→

Tier 3: Permanent Hardware Upgrade 1,400 – 10,000 USD

Honest Hardware Sizing for 70B @ 128k

Hardware Setup	Total VRAM	70B Q4 @ 128k FP8 (1 Stream)?	Verdict
Dual Used RTX 3090	48 GB	Only at 32k	Fits 32k with Q8_0 KV; crashes at 64k/128k.
Single RTX 5090	32 GB	No	Cannot load 70B weights without CPU paging.
Dual RTX 5090	64 GB	Yes, Tight	37.9 GB weights + 20 GB FP8 KV = 57.9 GB. Fits with ~4GB headroom.
NVIDIA RTX PRO 6000 Ada	48 GB / 96 GB	Yes (96GB Setup)	Flawless enterprise workstation headroom.
Apple Mac Studio (128GB Unified)	~96 GB Metal	Yes, Comfort	Runs full 128k Q4_K_M comfortably; prompt evaluation is slower than CUDA.

Test Your Exact Hardware & Model in Real Time

Our client-side VRAM calculator simulates token speeds, KV-cache headroom, and TCO break-even for over 20 open-weight models.

Launch Full GPU Garden Matchmaker →