# Fix bitsandbytes CUDA Setup Failed Despite GPU Being Available & NF4 4-Bit Quantization OOM

> **Direct Error Triage**: When HuggingFace Transformers, QLoRA (PEFT/Unsloth), or Diffusers throws RuntimeError: CUDA Setup failed despite GPU being available, bitsandbytes failed to locate a matching libcudart.so / cudart64_*.dll for your PyTorch CUDA version—or you are running an outdated wheel on Windows or Blackwell (sm_120). Fix it by upgrading to bitsandbytes>=0.45.3 (which ships native multi-backend Windows/Linux CUDA 11.8–12.8 binaries) and passing device_map='auto' with low_cpu_mem_usage=True.

## Error Signature
```
RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information
```

## Root Cause & Memory Allocation Mechanics
Historically, `bitsandbytes` (`load_in_4bit=True` NF4 and `load_in_8bit=True` LLM.int8()) dynamically searched `LD_LIBRARY_PATH` at import time to dlopen `libcudart.so`. Whenever a Conda virtualenv or Docker container mixed system CUDA toolkit paths (e.g., `/usr/local/cuda-12.1`) with PyTorch's bundled pip runtime (`nvidia-cuda-runtime-cu12`), `libbitsandbytes_cuda*.so` failed to link and raised `CUDA Setup failed despite GPU being available`. A second trap occurs during `BitsAndBytesConfig(load_in_4bit=True)` loading: if you load a 16-bit FP16 checkpoint to quantize on-the-fly without `device_map='auto'`, Transformers stages full FP16 weights into VRAM before compressing them to 4-bit NF4, causing an instant OOM on 12GB–16GB GPUs!

## Step-by-Step CLI Remediation
### Ollama
```bash
# Run pre-quantized 4-bit models directly in Ollama without Python bitsandbytes compilation
ollama run llama3.1:8b-instruct-q4_K_M
```

### Llama.cpp
```bash
# Upgrade bitsandbytes inside your project virtualenv and run its built-in diagnostics:
./.venv/bin/pip install -U "bitsandbytes>=0.45.3" "accelerate>=1.2.0" "transformers>=4.48.0"
./.venv/bin/python -m bitsandbytes
```

### vLLM
```bash
# Serve pre-quantized BitsAndBytes or AWQ checkpoints in vLLM without FP16 staging OOM:
vllm serve unsloth/Llama-3.1-8B-Instruct-bnb-4bit \
  --quantization bitsandbytes \
  --load-format bitsandbytes \
  --max-model-len 8192
```

## Verified Hardware & Cloud Pathways
- Explore 24GB GPUs on Amazon: https://www.amazon.com/s?k=RTX+4090+24GB+graphics+card&tag=sparecables-20 (rel="noopener sponsored")
- Rent on RunPod: https://runpod.io?ref=ynypzu94 (rel="noopener sponsored")
- Rent on Vast.ai: https://cloud.vast.ai/?ref_id=723130 (rel="noopener sponsored")

---
*Interactive Sizing Calculator available at: https://www.gpugarden.com/oom/bitsandbytes-cuda-setup-failed/*