Fix vLLM RuntimeError: Free memory on device is less than desired GPU memory utilization (--gpu-memory-utilization)

Root Cause & Memory Allocation Mechanics

Unlike relative memory flags that take 90% of *currently free* VRAM, vLLM's `--gpu-memory-utilization` parameter (default `0.90`) specifies a fraction of *total physical* GPU capacity (`total_gpu_memory * 0.90`). On a 24.0 GiB RTX 3090 or RTX 4090, `0.90` demands at least `21.60 GiB` of completely untouched VRAM at startup. If Ubuntu `gnome-shell`/`Xorg`, Windows Desktop Window Manager, or a lingering `VLLM::EngineCore` child process from a `Ctrl+C` interrupt is holding `2.8 GiB` (`21.2 GiB` free), vLLM's pre-flight check (`free_gpu_memory < total_gpu_memory * utilization`) immediately aborts with a `ValueError`.

Exact CLI Remediation Commands

# Inspect and unload background Ollama models occupying VRAM
ollama ps
ollama stop 
# Or stop the background service completely before launching vLLM:
sudo systemctl stop ollama

# Find and terminate zombie CUDA processes holding GPU memory on Linux
nvidia-smi
sudo fuser -v /dev/nvidia* | awk '{for(i=1;i<=NF;i++) if ($i ~ /^[0-9]+$/) print $i}' | xargs -r kill -9

# Lower --gpu-memory-utilization to match actual free VRAM after OS overhead
vllm serve model \
  --gpu-memory-utilization 0.82 \
  --max-model-len 8192 \
  --kv-cache-dtype fp8

Frequently Asked Questions