OLLAMA_KV_CACHE_TYPE Guide: Reduce Ollama VRAM Usage by 50% to 75%
OLLAMA_KV_CACHE_TYPE=q8_0 / OLLAMA_KV_CACHE_TYPE=q4_0Root Cause & Memory Allocation Mechanics
In autoregressive transformers, key and value tensors for every past token must be retained in memory for attention computation. In dense models like LLaMA 3.3 70B (80 layers, 8 KV heads, headDim 128), FP16 KV-cache takes 320 KB per token. At 32,768 context, KV-cache alone takes 10.0 GB; at 131,072 context, it takes 40.0 GB! Quantizing key and value tensors to 8-bit integers (`q8_0`) reduces storage to 170 KB/token with >99.8% reasoning fidelity.
Exact CLI Remediation Commands
# Linux / macOS Terminal
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_FLASH_ATTENTION=1
ollama serve
# Windows PowerShell 5.1
$env:OLLAMA_KV_CACHE_TYPE="q8_0"; $env:OLLAMA_FLASH_ATTENTION="1"; ollama serve
# Linux systemd Service (/etc/systemd/system/ollama.service)
[Service]
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_FLASH_ATTENTION=1"# Native llama.cpp equivalents
./llama-server -m model.gguf -ctk q8_0 -ctv q8_0 -fa on# Native vLLM FP8 KV Cache
vllm serve model --kv-cache-dtype fp8Upgrade VRAM on Amazon
If software quantization cannot bridge your memory deficit, explore 16GB–24GB GPUs.
Frequently Asked Questions
Does OLLAMA_KV_CACHE_TYPE=q8_0 reduce reasoning accuracy or code quality?
Empirical benchmarks show <0.2% perplexity change and 0.0% degradation on HumanEval coding tests. It is virtually lossless while halving memory footprint.
Can I use OLLAMA_KV_CACHE_TYPE=q4_0 on smaller 8B models?
Yes, but 4-bit KV quantization can introduce minor hallucinations on small models (<8B) during multi-hop reasoning. For 8B models, `q8_0` is recommended; for 70B+ models, `q4_0` works exceptionally well.
Verify Your Hardware Memory Math
Use the GPU Garden Matchmaker to calculate exact KV-cache footprints and avoid future OOM crashes.
Open Interactive Calculator →