Fix llama_kv_cache_init: failed to allocate buffer for kv cache in Llama.cpp

Root Cause & Memory Allocation Mechanics

Modern GGUF checkpoints (such as LLaMA 3.1/3.3, Qwen 2.5/3, and Mistral) declare `llama.context_length = 131072` inside their file header. In recent releases of `llama-cli` and `llama-server`, omitting `-c` (or passing `-c 0`) instructs `llama_kv_cache_init()` to allocate the full 131,072-token KV buffer in 16-bit FP16 upfront. On a 32-layer 8B model, 128k FP16 KV-cache consumes 16.0 GiB; on an 80-layer 70B model, it demands 40.0 GiB! Consequently, the GGUF weight tensors load cleanly into VRAM first, and the process crashes milliseconds later during `llama_new_context_with_model()` when the KV allocator requests tens of gigabytes of non-existent VRAM.

Exact CLI Remediation Commands

# Override 128k default context in Modelfile or API payload and quantize KV
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
# Inside Modelfile:
PARAMETER num_ctx 8192

# Explicitly clamp -c to 8192/16384 and compress K/V cache to 8-bit or 4-bit
./llama-server -m model.gguf -ngl 99 -c 16384 -np 1 -fa on -ctk q8_0 -ctv q8_0

# Ultra-low VRAM mode (75% KV reduction via q4_0)
./llama-server -m model.gguf -ngl 99 -c 32768 -np 1 -fa on -ctk q4_0 -ctv q4_0

# In vLLM, cap --max-model-len so KV cache blocks fit remaining VRAM
vllm serve model --max-model-len 16384 --kv-cache-dtype fp8 --gpu-memory-utilization 0.92

Frequently Asked Questions