How Many Concurrent Users Can One GPU Handle for LLMs?
Multi-Stream VRAM Sizing & Concurrency Math for vLLM & SGLang
🧮 Root Cause & Mathematical Reality
While static model weights are shared across all incoming requests, each active user stream requires its own independent KV cache allocation. Sizing a GPU server for weights alone causes out-of-memory crashes the moment multiple requests arrive simultaneously.
- Model weights are shared across infinite users; KV cache memory must be multiplied by active concurrent streams (N).
- On an 80GB A100 running 70B AWQ (40GB weights), 35GB usable KV cache supports 1 × 128k stream, 5 × 32k streams, or 20 × 8k streams.
- Formula: MaxStreams = floor((Usable_VRAM - Model_Weights - Drivers) / KV_per_stream).
🛠️ Actionable Fix & Tuning Steps
- Calculate per-stream KV cache requirement using: 2 × Layers × KV_Heads × Head_Dim × Bytes × Avg_Tokens.
- Subtract model weights and CUDA buffers from total VRAM to determine available KV cache memory pool.
- Divide the pool by per-stream KV cache to find theoretical maximum concurrent streams.
- Enable continuous batching and PagedAttention in vLLM to dynamically allocate token blocks.
- If demand exceeds local capacity, scale out with on-demand cloud workers on RunPod or Vast.ai.
⚖️ The 3-Tier Honesty Funnel
Enable FP8/Q8_0 KV-cache or clamp context tokens to fit 100% within your local GPU memory.
If model weights or 128k context exceed physical VRAM, rent an 80GB GPU by the hour.
❓ Frequently Asked Questions
How many users can an RTX 4090 (24GB) handle running LLaMA 3.1 8B?
At Q4, 8B weights take ~5.5 GB. With 16GB free for KV cache, an RTX 4090 can handle ~16 concurrent users at 8k context or ~32 users at 4k context.
What happens when more users connect than KV cache allows?
In vLLM, excess requests enter a priority queue. In engines without paging, the server crashes with CUDA OOM.
Explore the Full OOM Troubleshooting Hub
Master VRAM requirements for LLaMA 3.3 70B, DeepSeek V3, Qwen 2.5, and vLLM concurrency.
View All 12 GPU & VRAM Fixers →