Fix CUDA error: an illegal memory access was encountered in Ollama & Llama.cpp

Root Cause & Memory Allocation Mechanics

Because CUDA kernel launches are asynchronous, the GPU hardware exception (`cudaErrorIllegalAddress`, code 700) is almost always reported on the *next* synchronous call (`cudaStreamSynchronize` or `cudaMemcpy`), masking the true offending operation unless `CUDA_LAUNCH_BLOCKING=1` is exported. Software triggers include pairing experimental `q4_0` KV-cache quantization with non-standard head dimensions (`headDim = 80` or `256`) or multi-GPU PCIe P2P transfers across mismatched IOMMU groups. Hardware triggers are equally common: sustained matrix-multiplication bursts push GDDR6X junction temps past 102°C on RTX 3090 cards or cause microsecond power spikes that trip marginal PCIe riser cables and daisy-chained 8-pin PSU rails.

Exact CLI Remediation Commands

# 1. Isolate software vs hardware fault by disabling FlashAttention & KV quant temporarily
export OLLAMA_FLASH_ATTENTION=0
export OLLAMA_KV_CACHE_TYPE=f16
export CUDA_LAUNCH_BLOCKING=1
ollama serve

# 2. Disable memory-mapped weights and reduce micro-batch size (-ub 256)
./llama-server -m model.gguf -ngl 95 --no-mmap -b 512 -ub 256 -c 8192

# 3. Stabilize GPU power draw & GDDR6X thermals (run as Administrator/root):
nvidia-smi -pl 280

# Disable custom all-reduce and CUDA graphs to isolate multi-GPU P2P faults
export CUDA_LAUNCH_BLOCKING=1
vllm serve model --enforce-eager --disable-custom-all-reduce

Frequently Asked Questions