Fix ROCm HIP out of memory (hipErrorOutOfMemory) on AMD Radeon RX 7900 XTX, RX 9070 XT & Strix Halo
RuntimeError: HIP out of memory. Tried to allocate ... MiB (hipErrorOutOfMemory)Root Cause & Memory Allocation Mechanics
AMD's ROCm HIP runtime mirrors CUDA's memory model on discrete graphics cards (`gfx1100` RDNA 3 and `gfx1201` RDNA 4), but exhibits two AMD-specific pitfalls. First, ROCm's Triton and MIOpen kernel compilers allocate sizable scratch workspaces during first-run tuning unless `PYTORCH_HIP_ALLOC_CONF=expandable_segments:True,garbage_collection_threshold:0.8` is set. Second, on AMD Strix Halo 128GB unified-memory systems (`gfx1151`), the Linux `amdgpu` driver defaults Graphics Translation Table (`GTT`) memory to only 50% of system RAM (64GB), causing `hipErrorOutOfMemory` on 70B–120B models even when 60GB of system LPDDR5X is sitting idle!
Exact CLI Remediation Commands
# Ensure correct GFX target override (if needed) and enable 8-bit KV cache on ROCm
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
# For Strix Halo (gfx1151) / RDNA 3 (gfx1100):
ollama serve# On Strix Halo 128GB Linux, unlock 110GB GTT unified memory in /etc/default/grub:
# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.gttsize=112640 ttm.pages_limit=28835840"
./llama-server -m model.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0# Optimize PyTorch HIP allocator fragmentation and enable ROCm FlashAttention
export PYTORCH_HIP_ALLOC_CONF=expandable_segments:True,max_split_size_mb:128
export VLLM_USE_TRITON_FLASH_ATTN=1
vllm serve model --dtype float16 --max-model-len 8192 --gpu-memory-utilization 0.90Upgrade VRAM on Amazon
If software quantization cannot bridge your memory deficit, explore 16GB–24GB GPUs.
Frequently Asked Questions
How do I check how much VRAM and GTT memory ROCm sees on my AMD GPU?
Run `rocm-smi --showmeminfo vram gtt` or `amdgpu_top`. On Strix Halo APUs, verify that `gtt total` reflects your expanded kernel limit (e.g., 96GB–110GB) rather than a tiny 512MB–2GB BIOS UMA carve-out.
Should I set a huge 96GB UMA Frame Buffer in BIOS on AMD Strix Halo?
On Linux, leaving BIOS UMA on `Auto` (or `4GB`) and expanding `amdgpu.gttsize` + `ttm.pages_limit` is superior because memory remains dynamically shared between CPU and GPU instead of permanently locked away from the OS.
Verify Your Hardware Memory Math
Use the GPU Garden Matchmaker to calculate exact KV-cache footprints and avoid future OOM crashes.
Open Interactive Calculator →