# Fix vLLM NCCL Error & Custom Allreduce Failed with --tensor-parallel-size 2 on Dual RTX 3090 / 4090

> **Direct Error Triage**: Running vLLM with --tensor-parallel-size 2 (or 4) on Dual RTX 3090, RTX 4090, or RTX 5090 workstations frequently crashes with RuntimeError: NCCL error: unhandled system error or P2P access not supported between devices. NVIDIA disables direct PCIe Peer-to-Peer (P2P) BAR1 transfers on consumer GeForce drivers when AMD/Intel IOMMU is enabled or when cards sit across chipset PCIe switches. Fix it immediately by exporting NCCL_P2P_DISABLE=1 and passing --disable-custom-all-reduce.

## Error Signature
```
RuntimeError: NCCL error: unhandled system error / P2P access not supported between devices
```

## Root Cause & Memory Allocation Mechanics
Tensor Parallelism (`--tensor-parallel-size 2`) splits every individual transformer matrix multiplication horizontally across both GPUs, requiring an `AllReduce` synchronization collective twice per layer (160 times per token on an 80-layer 70B model). By default, NVIDIA's NCCL library and vLLM's custom P2P all-reduce kernel attempt to map GPU 1's PCIe BAR memory directly into GPU 0's virtual address space. On consumer AM5, LGA1700, and Threadripper motherboards with `IOMMU = Enabled` (or Access Control Services `ACS` active on PCIe root ports), the Linux kernel blocks direct peer DMA—causing NCCL to hang or crash with `unhandled system error` unless communications are routed through shared host pinned memory (`SHM`).

## Step-by-Step CLI Remediation
### Ollama
```bash
# Ollama uses pipeline layer splitting automatically without NCCL P2P issues
export CUDA_VISIBLE_DEVICES=0,1
ollama run llama3.3:70b
```

### Llama.cpp
```bash
# llama.cpp layer split (-sm layer) transfers tiny 16KB activations without needing P2P
./llama-server -m llama-3.3-70b-q4_k_m.gguf -ngl 99 -sm layer -fa on -ctk q8_0 -ctv q8_0
```

### vLLM
```bash
# Bypass broken consumer PCIe P2P in NCCL and disable vLLM custom P2P all-reduce
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1
export CUDA_DEVICE_MAX_CONNECTIONS=1
vllm serve meta-llama/Llama-3.3-70B-Instruct-AWQ \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92
```

## Verified Hardware & Cloud Pathways
- Explore 24GB GPUs on Amazon: https://www.amazon.com/s?k=RTX+4090+24GB+graphics+card&tag=sparecables-20 (rel="noopener sponsored")
- Rent on RunPod: https://runpod.io?ref=ynypzu94 (rel="noopener sponsored")
- Rent on Vast.ai: https://cloud.vast.ai/?ref_id=723130 (rel="noopener sponsored")

---
*Interactive Sizing Calculator available at: https://www.gpugarden.com/oom/tensor-parallel-size-nccl-error/*