# GPU Garden — Full LLM Hardware & VRAM Memory Specification (llms-full.txt)

> Authoritative, zero-maintenance, client-side technical specification, memory formulas, context window scaling tables, and hardware requirement benchmarks for open-weight AI inference.
> Canonical URL: https://gpugarden.com/llms-full.txt
> Summary Reference: https://gpugarden.com/llms.txt
> Interactive Matchmaker: https://gpugarden.com/

---

## 1. Mathematical Memory Formulation

Accurate calculation of total VRAM allocation prevents Out-Of-Memory (OOM) runtime crashes and GPU kernel panics:

$$\text{Total VRAM (GB)} = \text{Model Weights} + \text{KV-Cache Memory} + \text{Peak Activation Buffer} + \text{CUDA Runtime / OS Overhead}$$

### 1.1 Model Weights Memory
$$\text{Weights (GB)} = P \times \left(\frac{\text{BPW}}{8}\right) \times \mu$$

Where:
* $P$ = Parameter count in billions.
* $\text{BPW}$ = Bits Per Weight for the selected quantization scheme.
* $\mu$ = Quantization format packaging overhead factor (scale factors, block headers, padding).

| Precision / Format | Average BPW | Overhead Factor ($\mu$) | Quality Retention vs FP16 | Typical Use Case |
| :--- | :--- | :--- | :--- | :--- |
| **FP16 / BF16 (Full Precision)** | 16.00 | 1.02 | 100.0% (Baseline) | Model training, fine-tuning, ground truth benchmarks |
| **Q8_0 (8-bit Quantization)** | 8.50 | 1.03 | 99.9% Perplexity | Datacenter inference, near-lossless evaluation |
| **Q6_K (6-bit GGUF)** | 6.56 | 1.04 | 99.5% Perplexity | High-capacity consumer workstation (48GB rigs) |
| **Q5_K_M (5-bit GGUF)** | 5.50 | 1.04 | 99.0% Perplexity | Balanced sweet spot for 16GB–24GB GPUs |
| **Q4_K_M (4-bit GGUF)** | 4.50 | 1.05 | 98.6% Perplexity | Universal community standard; lowest VRAM with high reasoning |
| **AWQ 4-Bit** | 4.25 | 1.06 | 98.2% Perplexity | Optimized for vLLM & TensorRT-LLM batch serving |
| **GPTQ 4-Bit** | 4.25 | 1.06 | 98.0% Perplexity | Fast Linux CUDA kernel inference (ExLlamaV2) |
| **Q3_K_M (3-bit GGUF)** | 3.50 | 1.07 | 93.5% Perplexity | Extreme fit on memory-constrained budget hardware |

### 1.2 Key-Value (KV) Cache Memory
The Key-Value attention cache grows strictly with sequence length, batch size, and transformer layer depth:

#### Standard Multi-Head Attention (MHA) & Grouped-Query Attention (GQA):
$$\text{KV-Cache (GiB)} = \frac{2 \times L \times K \times D \times S \times B \times \beta}{1024^3}$$

Where:
* $L$ = Number of transformer layers.
* $K$ = Number of Key-Value attention heads ($K = 8$ on LLaMA 3.1/3.3 70B, $K = 8$ on LLaMA 8B).
* $D$ = Dimension per attention head (typically 128).
* $S$ = Sequence length / context token count (e.g., 8,192 to 131,072).
* $B$ = Concurrent batch size (default $B = 1$ for single-user local inference).
* $\beta$ = KV cache precision bytes per token:
  * $\beta = 2.0$ bytes for FP16 precision.
  * $\beta = 1.0$ byte for FP8 precision (supported natively on Ada Lovelace / RTX 40 and Hopper / H100).
  * $\beta = 0.5$ bytes for Q4 quantization.

#### Multi-Head Latent Attention (MLA - DeepSeek V2 / V3):
$$\text{MLA KV-Cache (GiB)} = \frac{2 \times L \times D_{\text{latent}} \times S \times B \times \beta}{1024^3}$$
DeepSeek projects keys and values into a compressed latent space ($D_{\text{latent}} = 512$), reducing KV-cache VRAM consumption by 75–85% compared to conventional MHA.

### 1.3 Peak Activation Buffer
$$\text{Activation (GB)} = \max\left(0.40\text{ GB}, \frac{S}{32768} \times \frac{H}{4096} \times 0.70\text{ GB}\right)$$
Where $H = D \times \text{Query\_Heads}$ (hidden dimension).

### 1.4 CUDA Runtime & OS Video Display Buffer
* **NVIDIA Desktop GPUs (with Display Attached)**: 1.20–1.60 GB reserved for Windows DWM / Linux X11/Wayland and CUDA runtime contexts.
* **NVIDIA Headless / Secondary Compute GPUs**: 0.60–0.80 GB reserved.
* **Apple Silicon Metal**: 0.00 GB CUDA reservation. However, macOS imposes a default single-process Metal allocation threshold of ~75% of total unified memory unless overridden via `sysctl iogpu.wired_mem_limit`.

---

## 2. Context Window Scaling Reference Matrix

How context window tokens expand KV-cache VRAM across precisions (FP16 vs FP8 vs Q4) for Batch Size 1:

### 2.1 Meta LLaMA 3.3 70B (80 Layers, 8 KV Heads, Head Dim 128)
| Context Tokens | FP16 KV-Cache | FP8 KV-Cache | Q4 KV-Cache | Total VRAM (Q4_K_M Weights + KV) |
| :--- | :--- | :--- | :--- | :--- |
| **2,048 Tokens (2k)** | 0.52 GB | 0.26 GB | 0.13 GB | 42.54 GB |
| **4,096 Tokens (4k)** | 1.05 GB | 0.52 GB | 0.26 GB | 43.07 GB |
| **8,192 Tokens (8k - Standard)** | 2.10 GB | 1.05 GB | 0.52 GB | 44.12 GB |
| **16,384 Tokens (16k)** | 4.20 GB | 2.10 GB | 1.05 GB | 46.22 GB |
| **32,768 Tokens (32k)** | 8.40 GB | 4.20 GB | 2.10 GB | 50.42 GB |
| **65,536 Tokens (64k)** | 16.80 GB | 8.40 GB | 4.20 GB | 58.82 GB |
| **131,072 Tokens (128k - Native Max)** | 33.60 GB | 16.80 GB | 8.40 GB | 75.62 GB |

### 2.2 Alibaba Qwen 2.5 Coder 32B (64 Layers, 8 KV Heads, Head Dim 128)
| Context Tokens | FP16 KV-Cache | FP8 KV-Cache | Q4 KV-Cache | Total VRAM (Q4_K_M Weights + KV) |
| :--- | :--- | :--- | :--- | :--- |
| **2,048 Tokens (2k)** | 0.42 GB | 0.21 GB | 0.10 GB | 19.92 GB |
| **4,096 Tokens (4k)** | 0.84 GB | 0.42 GB | 0.21 GB | 20.34 GB |
| **8,192 Tokens (8k - Standard)** | 1.68 GB | 0.84 GB | 0.42 GB | 21.60 GB (RTX 4090 Sweet Spot) |
| **16,384 Tokens (16k)** | 3.36 GB | 1.68 GB | 0.84 GB | 23.28 GB (Tight on 24GB) |
| **32,768 Tokens (32k)** | 6.71 GB | 3.36 GB | 1.68 GB | 26.63 GB (Requires 32GB+) |
| **65,536 Tokens (64k)** | 13.42 GB | 6.71 GB | 3.36 GB | 33.34 GB (Requires 48GB) |
| **131,072 Tokens (128k - Native Max)** | 26.84 GB | 13.42 GB | 6.71 GB | 46.76 GB (Requires 48GB rig) |

### 2.3 Meta LLaMA 3.1 8B (32 Layers, 8 KV Heads, Head Dim 128)
| Context Tokens | FP16 KV-Cache | FP8 KV-Cache | Q4 KV-Cache | Total VRAM (Q4_K_M Weights + KV) |
| :--- | :--- | :--- | :--- | :--- |
| **4,096 Tokens (4k)** | 0.42 GB | 0.21 GB | 0.10 GB | 6.42 GB (Fits on 8GB GPU) |
| **8,192 Tokens (8k - Standard)** | 0.84 GB | 0.42 GB | 0.21 GB | 7.60 GB (Fits on 8GB GPU) |
| **16,384 Tokens (16k)** | 1.68 GB | 0.84 GB | 0.42 GB | 8.44 GB (Requires 12GB) |
| **32,768 Tokens (32k)** | 3.36 GB | 1.68 GB | 0.84 GB | 10.12 GB (Fits on 12GB RTX 3060) |
| **65,536 Tokens (64k)** | 6.71 GB | 3.36 GB | 1.68 GB | 13.47 GB (Fits on 16GB GPU) |
| **131,072 Tokens (128k - Native Max)** | 13.42 GB | 6.71 GB | 3.36 GB | 20.18 GB (Fits on 24GB GPU) |

---

## 3. Comprehensive Open-Weight Model Sizing Catalog

Standard baseline calculations evaluated at **8,192 tokens context window** with 1.50 GB activation/CUDA driver buffer:

| Model Architecture | Parameters | 4-Bit (Q4_K_M) | 8-Bit (Q8_0) | 16-Bit (FP16) | Minimum Hardware Sweet Spot | Canonical Guide URL |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **Meta LLaMA 3.3 70B Instruct** | 70.6B Dense | 44.12 GB | 79.14 GB | 148.50 GB | Dual RTX 3090 (48GB) / Mac 64GB | https://gpugarden.com/can-i-run-llama-3-3-70b/ |
| **DeepSeek V3 671B MoE** | 671B (37B active) | ~380.0 GB | ~670.0 GB | ~1,340.0 GB | 8x H100 80GB SXM5 Cloud Node | https://gpugarden.com/can-i-run-deepseek-v3/ |
| **Alibaba Qwen 2.5 Coder 32B** | 32.5B Dense | 21.60 GB | 37.60 GB | 69.80 GB | NVIDIA GeForce RTX 4090 (24GB) | https://gpugarden.com/rtx-4090-local-ai-guide/ |
| **Alibaba Qwen 2.5 72B Instruct** | 72.7B Dense | 45.40 GB | 81.40 GB | 152.90 GB | Dual RTX 3090 (48GB) / Mac 64GB | https://gpugarden.com/vram/llama-3-3-70b/ |
| **Alibaba Qwen 2.5 14B Instruct** | 14.7B Dense | 10.90 GB | 18.20 GB | 32.90 GB | NVIDIA RTX 4070 Ti Super (16GB) | https://gpugarden.com/ |
| **Mistral Small 24B Instruct** | 24.1B Dense | 16.80 GB | 28.40 GB | 53.20 GB | NVIDIA RTX 4090 24GB / RTX 3090 | https://gpugarden.com/rtx-4090-local-ai-guide/ |
| **Meta LLaMA 3.1 8B Instruct** | 8.03B Dense | 7.60 GB | 11.50 GB | 19.40 GB | NVIDIA RTX 4060 (8GB) / 3060 (12GB) | https://gpugarden.com/vram/llama-3-1-8b/ |
| **Meta LLaMA 3.2 11B Vision** | 13.8B Multimodal | 10.50 GB | 17.10 GB | 30.70 GB | NVIDIA RTX 4070 (12GB) / 16GB | https://gpugarden.com/ |
| **Meta LLaMA 3.2 3B Instruct** | 3.21B Dense | 4.30 GB | 5.80 GB | 9.00 GB | Entry 6GB–8GB GPUs / Laptops | https://gpugarden.com/ |
| **Meta LLaMA 3.2 1B Instruct** | 1.23B Dense | 2.50 GB | 3.10 GB | 4.30 GB | Budget 4GB GPUs / Edge Devices | https://gpugarden.com/ |
| **DeepSeek Coder V2 Lite** | 15.7B (2.4B active) | 12.00 GB | 19.80 GB | 35.80 GB | NVIDIA RTX 4070 Ti Super (16GB) | https://gpugarden.com/ |
| **FLUX.1 [dev] Image DiT** | 12.0B DiT Flow | 12.00 GB (NF4) | 18.50 GB | 26.50 GB | NVIDIA RTX 3060 12GB / RTX 4090 24GB | https://gpugarden.com/vram/flux-1-dev/ |
| **Stable Diffusion XL 1.0 (SDXL)**| 2.6B UNet | 6.80 GB (Q4) | 8.20 GB | 11.50 GB | NVIDIA RTX 3060 (12GB) / RTX 4060 (8GB) | https://gpugarden.com/ |

---

## 4. Hardware Sizing & Memory Bandwidth Reference

Token generation throughput during the autoregressive decoding phase is fundamentally **memory bandwidth bound**:

$$\text{Throughput (tokens/sec)} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Active Model Memory Footprint (GB)}}$$

| GPU Hardware Platform | VRAM Capacity | Memory Bus | Bandwidth | Peak LLM Speed (Q4) | Primary Sweet Spot Workload | Typical Street Price |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **NVIDIA GeForce RTX 4090** | 24 GB GDDR6X | 384-bit | 1,008 GB/s | ~42 tok/s (32B Q4) | Fastest desktop card; Qwen 32B sweet spot | $1,750 – $1,950 (New) |
| **Dual NVIDIA RTX 3090** | 48 GB GDDR6X | $2 \times 384$-bit | 1,872 GB/s | ~25 tok/s (70B Q4) | Undisputed king of 70B homelabs | $1,300 – $1,500 (Used total) |
| **NVIDIA GeForce RTX 3090** | 24 GB GDDR6X | 384-bit | 936 GB/s | ~34 tok/s (32B Q4) | Best single-card value champion | $650 – $750 (Used) |
| **Apple M3 / M4 Max (64GB)** | 64 GB Unified | 512-bit | 300–400 GB/s | ~15 tok/s (70B Q4) | Silent, low-power 48GB allocatable workstation | $3,400+ (Mac Studio/Book) |
| **Apple M2 / M3 Ultra (128GB)**| 128 GB Unified | 1024-bit | 800 GB/s | ~18 tok/s (70B FP16)| Large single-pool unified workstation | $3,999+ (Mac Studio) |
| **AMD Radeon RX 7900 XTX** | 24 GB GDDR6 | 384-bit | 960 GB/s | ~32 tok/s (32B Q4) | Sub-$1,000 24GB card (Linux ROCm) | $899 – $999 (New) |
| **NVIDIA RTX 4070 Ti Super** | 16 GB GDDR6X | 256-bit | 672 GB/s | ~45 tok/s (14B Q4) | Modern 16GB efficiency leader | $799 – $849 (New) |
| **NVIDIA GeForce RTX 3060** | 12 GB GDDR6 | 192-bit | 360 GB/s | ~38 tok/s (8B Q4) | Entry-level budget hero (~$280) | $260 – $290 (New) |
| **NVIDIA GeForce RTX 4060** | 8 GB GDDR6 | 128-bit | 272 GB/s | ~42 tok/s (3B Q4) | Fast Ada cores, strictly capped by 8GB VRAM | $299 – $329 (New) |
| **NVIDIA A100 80GB SXM4 (Cloud)**| 80 GB HBM2e | 5120-bit | 2,039 GB/s | ~45 tok/s (70B Q8) | Cloud rental pod standard on RunPod/Vast.ai | $0.80 – $1.20 / hour |
| **NVIDIA H100 80GB SXM5 (Cloud)**| 80 GB HBM3 | 5120-bit | 3,350 GB/s | ~70 tok/s (70B Q8) | Frontier datacenter benchmark | $2.50 – $4.50 / hour |

---

## 5. Production CLI Execution Commands

### 5.1 Meta LLaMA 3.3 70B Instruct
* **Ollama (Multi-GPU Split)**:
  ```bash
  ollama run llama3.3:70b-instruct-q4_K_M
  ```
* **vLLM (Tensor Parallelism over 2 GPUs with FP8 KV-Cache)**:
  ```bash
  vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.95
  ```
* **llama.cpp (Row-Split Dual 3090 Rig)**:
  ```bash
  llama-server -m llama-3.3-70b-instruct.Q4_K_M.gguf -ngl 99 -sm row -c 8192 --cache-type-k q8_0 --cache-type-v q8_0
  ```

### 5.2 Alibaba Qwen 2.5 Coder 32B on RTX 4090 24GB
* **Ollama (Native 24GB Fit)**:
  ```bash
  ollama run qwen2.5-coder:32b-instruct-q4_K_M
  ```
* **vLLM (High-Throughput Local Coding Assistant)**:
  ```bash
  vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
    --max-model-len 8192 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90
  ```

### 5.3 DeepSeek V3 671B MoE on 8x H100 SXM5 Cluster
* **vLLM (Tensor Parallel 8)**:
  ```bash
  vllm serve deepseek-ai/DeepSeek-V3 \
    --tensor-parallel-size 8 \
    --trust-remote-code \
    --max-model-len 8192
  ```
* **SGLang (Optimized MLA Implementation)**:
  ```bash
  python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --trust-remote-code
  ```

---

## 6. Cloud vs Local True Cost of Ownership (TCO) Economics

### 6.1 Mathematical TCO Formulation
$$\text{Cost}_{\text{Local}} = \text{Hardware CapEx} + \left(\text{Hours} \times \frac{\text{TDP (Watts)}}{1000} \times \text{Electricity Rate (\$/kWh)}\right) + \text{Depreciation}$$
$$\text{Cost}_{\text{Cloud}} = \text{Hours} \times \text{Rental Rate (\$/hr)}$$

### 6.2 Decision Matrix
1. **Model Size $\le$ 32B (Qwen 32B, LLaMA 8B, Mistral 24B)**:
   * **Verdict: BUY LOCAL HARDWARE**.
   * A single RTX 4090 ($1,800) or used RTX 3090 ($700) provides 100% private, zero-latency, unmetered access that amortizes within 4 to 8 months against cloud hosting.
2. **Model Size 70B (LLaMA 3.3 70B, Qwen 72B)**:
   * **If running $>20$ hours/week**: Build a Dual RTX 3090 48GB rig (~$1,400 total).
   * **If running $<15$ hours/week**: Rent an 80GB A100 on RunPod ($0.89/hr).
3. **Model Size 671B MoE (DeepSeek V3)**:
   * **Verdict: ALWAYS RENT CLOUD COMPUTE**.
   * Local 8x H100 cluster CapEx is $325,000+ with 10.2 kW electrical infrastructure. Cloud rental on RunPod / Vast.ai ($2.50–$4.50/hr) saves over $320,000 upfront.

---

## 7. Canonical Programmatic URLs & Direct Landing Pages

* **Can I Run Meta LLaMA 3.3 70B?**: `https://gpugarden.com/can-i-run-llama-3-3-70b/`
* **Can I Run DeepSeek V3 671B MoE?**: `https://gpugarden.com/can-i-run-deepseek-v3/`
* **NVIDIA RTX 4090 24GB Local AI Guide**: `https://gpugarden.com/rtx-4090-local-ai-guide/`
* **RTX 3090 vs RTX 4090 Head-to-Head**: `https://gpugarden.com/compare/rtx-3090-vs-rtx-4090/`
* **Dual RTX 3090 vs RTX 4090 (48GB vs 24GB)**: `https://gpugarden.com/compare/dual-rtx-3090-vs-rtx-4090/`
* **RTX 3060 vs RTX 4060 (12GB vs 8GB VRAM Trap)**: `https://gpugarden.com/compare/rtx-3060-vs-rtx-4060/`
* **RTX 4070 Ti Super vs RTX 4080 Super**: `https://gpugarden.com/compare/rtx-4070-ti-super-vs-rtx-4080-super/`
* **Apple M3 Max vs RTX 4090**: `https://gpugarden.com/compare/apple-m3-max-vs-rtx-4090/`
* **Meta LLaMA 3.3 70B VRAM Profile**: `https://gpugarden.com/vram/llama-3-3-70b/`
* **Alibaba Qwen 2.5 Coder 32B VRAM Profile**: `https://gpugarden.com/vram/qwen-2-5-coder-32b/`
* **DeepSeek V3 VRAM Profile**: `https://gpugarden.com/vram/deepseek-v3/`
* **Meta LLaMA 3.1 8B VRAM Profile**: `https://gpugarden.com/vram/llama-3-1-8b/`
* **FLUX.1 [dev] VRAM Profile**: `https://gpugarden.com/vram/flux-1-dev/`