# Fix CUDA error: an illegal memory access was encountered in Ollama & Llama.cpp

> **Direct Error Triage**: Unlike a clean out-of-memory rejection, CUDA error: an illegal memory access was encountered (cudaErrorIllegalAddress) means a running GPU kernel attempted to dereference an out-of-bounds, unmapped, or corrupted device pointer—essentially a GPU segmentation fault. In Ollama and llama.cpp, this is triggered by three main culprits: unstable VRAM overclocks / PSU transient voltage droop under tensor load, a buggy FlashAttention/quantized KV kernel combination, or silent silent page-table eviction near 99% VRAM capacity.

## Error Signature
```
CUDA error: an illegal memory access was encountered (cudaErrorIllegalAddress)
```

## Root Cause & Memory Allocation Mechanics
Because CUDA kernel launches are asynchronous, the GPU hardware exception (`cudaErrorIllegalAddress`, code 700) is almost always reported on the *next* synchronous call (`cudaStreamSynchronize` or `cudaMemcpy`), masking the true offending operation unless `CUDA_LAUNCH_BLOCKING=1` is exported. Software triggers include pairing experimental `q4_0` KV-cache quantization with non-standard head dimensions (`headDim = 80` or `256`) or multi-GPU PCIe P2P transfers across mismatched IOMMU groups. Hardware triggers are equally common: sustained matrix-multiplication bursts push GDDR6X junction temps past 102°C on RTX 3090 cards or cause microsecond power spikes that trip marginal PCIe riser cables and daisy-chained 8-pin PSU rails.

## Step-by-Step CLI Remediation
### Ollama
```bash
# 1. Isolate software vs hardware fault by disabling FlashAttention & KV quant temporarily
export OLLAMA_FLASH_ATTENTION=0
export OLLAMA_KV_CACHE_TYPE=f16
export CUDA_LAUNCH_BLOCKING=1
ollama serve
```

### Llama.cpp
```bash
# 2. Disable memory-mapped weights and reduce micro-batch size (-ub 256)
./llama-server -m model.gguf -ngl 95 --no-mmap -b 512 -ub 256 -c 8192

# 3. Stabilize GPU power draw & GDDR6X thermals (run as Administrator/root):
nvidia-smi -pl 280
```

### vLLM
```bash
# Disable custom all-reduce and CUDA graphs to isolate multi-GPU P2P faults
export CUDA_LAUNCH_BLOCKING=1
vllm serve model --enforce-eager --disable-custom-all-reduce
```

## Verified Hardware & Cloud Pathways
- Explore 24GB GPUs on Amazon: https://www.amazon.com/s?k=RTX+4090+24GB+graphics+card&tag=sparecables-20 (rel="noopener sponsored")
- Rent on RunPod: https://runpod.io?ref=ynypzu94 (rel="noopener sponsored")
- Rent on Vast.ai: https://cloud.vast.ai/?ref_id=723130 (rel="noopener sponsored")

---
*Interactive Sizing Calculator available at: https://www.gpugarden.com/oom/ollama-cuda-error-an-illegal-memory-access/*