Fix vLLM NCCL Error & Custom Allreduce Failed with --tensor-parallel-size 2 on Dual RTX 3090 / 4090

Root Cause & Memory Allocation Mechanics

Tensor Parallelism (`--tensor-parallel-size 2`) splits every individual transformer matrix multiplication horizontally across both GPUs, requiring an `AllReduce` synchronization collective twice per layer (160 times per token on an 80-layer 70B model). By default, NVIDIA's NCCL library and vLLM's custom P2P all-reduce kernel attempt to map GPU 1's PCIe BAR memory directly into GPU 0's virtual address space. On consumer AM5, LGA1700, and Threadripper motherboards with `IOMMU = Enabled` (or Access Control Services `ACS` active on PCIe root ports), the Linux kernel blocks direct peer DMA—causing NCCL to hang or crash with `unhandled system error` unless communications are routed through shared host pinned memory (`SHM`).

Exact CLI Remediation Commands

# Ollama uses pipeline layer splitting automatically without NCCL P2P issues
export CUDA_VISIBLE_DEVICES=0,1
ollama run llama3.3:70b

# llama.cpp layer split (-sm layer) transfers tiny 16KB activations without needing P2P
./llama-server -m llama-3.3-70b-q4_k_m.gguf -ngl 99 -sm layer -fa on -ctk q8_0 -ctv q8_0

# Bypass broken consumer PCIe P2P in NCCL and disable vLLM custom P2P all-reduce
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1
export CUDA_DEVICE_MAX_CONNECTIONS=1
vllm serve meta-llama/Llama-3.3-70B-Instruct-AWQ \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92

Frequently Asked Questions