Fix RuntimeError: CUDA out of memory. Tried to allocate in PyTorch, vLLM & ComfyUI

Root Cause & Memory Allocation Mechanics

Unlike direct driver calls, PyTorch manages GPU memory through an internal caching allocator (`c10::cuda::CUDACachingAllocator`) that retains freed blocks in reserved pools to avoid expensive `cudaMalloc` synchronization overhead. During dynamic-shape workloads—such as variable-length LLM prompts in Transformers/vLLM or high-resolution latent passes in ComfyUI—the allocator frequently splits 20 MiB–512 MiB blocks into non-adjacent fragments. When an attention projection or 3D convolution subsequently demands a single contiguous 1,024 MiB tensor, `cudaMalloc` fails even if `reserved - allocated` reports 2.5 GiB of cached slack. Enabling virtual memory `expandable_segments:True` (supported on Linux and Windows CUDA 12.1+) stitches non-contiguous physical pages into a single virtual tensor address space without copying.

Exact CLI Remediation Commands

# Limit concurrent parallel request slots and enforce 8-bit KV cache
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve

# Disable host memory mapping spike and cap batch scratchpad size
./llama-server -m model-q4_k_m.gguf -ngl 99 -c 8192 -b 512 -ub 256 -fa on -ctk q8_0 -ctv q8_0

# Enable PyTorch expandable virtual segments and clamp max batched tokens
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:128
vllm serve model --dtype auto --kv-cache-dtype fp8 --max-model-len 8192 --max-num-seqs 8 --enforce-eager

Frequently Asked Questions