Fix ggml_backend_cuda_buffer_type_alloc_ctx: failed to allocate buffer in Llama.cpp & Ollama

Root Cause & Memory Allocation Mechanics

When the CUDA runtime executes `cudaMalloc` or `cudaMallocHost`, the NVIDIA driver verifies that contiguous unfragmented memory exists. If Windows Desktop Window Manager (DWM) or background Chrome tabs reserve 1.5 GB of VRAM, an allocation for a 22.8 GB model on a 24GB card will fail with error code 2 (`cudaErrorMemoryAllocation`). Additionally, memory fragmentation from repeated model reloads without restarting the server daemon can trigger this error even when total reported free VRAM appears sufficient.

Exact CLI Remediation Commands

# Option A: Force FP8 KV-cache
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama run model:q4_K_M

# Option B: Reduce context window via Modelfile
PARAMETER num_ctx 4096

# 1. Reduce offloaded layers from 99 to 32
./llama-server -m model.gguf -ngl 32 -c 4096

# 2. Enable Flash Attention & Quantized KV-cache
./llama-server -m model.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0

# Cap GPU memory utilization and enforce FP8 KV
vllm serve model --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 --max-model-len 8192

Frequently Asked Questions