# Fix RuntimeError: CUDA out of memory. Tried to allocate in PyTorch, vLLM & ComfyUI

> **Direct Error Triage**: PyTorch raises RuntimeError: CUDA out of memory when a tensor operation requests a contiguous device block that exceeds remaining unreserved VRAM or fragments the caching allocator pool. Resolve it immediately by setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, switching model weights from FP16/BF16 to 4-bit AWQ/GPTQ/GGUF or FP8, enabling gradient/activation checkpointing, or upgrading to a 24GB RTX 3090 / 4090.

## Error Signature
```
RuntimeError: CUDA out of memory. Tried to allocate ... MiB
```

## Root Cause & Memory Allocation Mechanics
Unlike direct driver calls, PyTorch manages GPU memory through an internal caching allocator (`c10::cuda::CUDACachingAllocator`) that retains freed blocks in reserved pools to avoid expensive `cudaMalloc` synchronization overhead. During dynamic-shape workloads—such as variable-length LLM prompts in Transformers/vLLM or high-resolution latent passes in ComfyUI—the allocator frequently splits 20 MiB–512 MiB blocks into non-adjacent fragments. When an attention projection or 3D convolution subsequently demands a single contiguous 1,024 MiB tensor, `cudaMalloc` fails even if `reserved - allocated` reports 2.5 GiB of cached slack. Enabling virtual memory `expandable_segments:True` (supported on Linux and Windows CUDA 12.1+) stitches non-contiguous physical pages into a single virtual tensor address space without copying.

## Step-by-Step CLI Remediation
### Ollama
```bash
# Limit concurrent parallel request slots and enforce 8-bit KV cache
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve
```

### Llama.cpp
```bash
# Disable host memory mapping spike and cap batch scratchpad size
./llama-server -m model-q4_k_m.gguf -ngl 99 -c 8192 -b 512 -ub 256 -fa on -ctk q8_0 -ctv q8_0
```

### vLLM
```bash
# Enable PyTorch expandable virtual segments and clamp max batched tokens
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:128
vllm serve model --dtype auto --kv-cache-dtype fp8 --max-model-len 8192 --max-num-seqs 8 --enforce-eager
```

## Verified Hardware & Cloud Pathways
- Explore 24GB GPUs on Amazon: https://www.amazon.com/s?k=RTX+4090+24GB+graphics+card&tag=sparecables-20 (rel="noopener sponsored")
- Rent on RunPod: https://runpod.io?ref=ynypzu94 (rel="noopener sponsored")
- Rent on Vast.ai: https://cloud.vast.ai/?ref_id=723130 (rel="noopener sponsored")

---
*Interactive Sizing Calculator available at: https://www.gpugarden.com/oom/cuda-out-of-memory/*