Fix bitsandbytes CUDA Setup Failed Despite GPU Being Available & NF4 4-Bit Quantization OOM

Root Cause & Memory Allocation Mechanics

Historically, `bitsandbytes` (`load_in_4bit=True` NF4 and `load_in_8bit=True` LLM.int8()) dynamically searched `LD_LIBRARY_PATH` at import time to dlopen `libcudart.so`. Whenever a Conda virtualenv or Docker container mixed system CUDA toolkit paths (e.g., `/usr/local/cuda-12.1`) with PyTorch's bundled pip runtime (`nvidia-cuda-runtime-cu12`), `libbitsandbytes_cuda*.so` failed to link and raised `CUDA Setup failed despite GPU being available`. A second trap occurs during `BitsAndBytesConfig(load_in_4bit=True)` loading: if you load a 16-bit FP16 checkpoint to quantize on-the-fly without `device_map='auto'`, Transformers stages full FP16 weights into VRAM before compressing them to 4-bit NF4, causing an instant OOM on 12GB–16GB GPUs!

Exact CLI Remediation Commands

# Run pre-quantized 4-bit models directly in Ollama without Python bitsandbytes compilation
ollama run llama3.1:8b-instruct-q4_K_M

# Upgrade bitsandbytes inside your project virtualenv and run its built-in diagnostics:
./.venv/bin/pip install -U "bitsandbytes>=0.45.3" "accelerate>=1.2.0" "transformers>=4.48.0"
./.venv/bin/python -m bitsandbytes

# Serve pre-quantized BitsAndBytes or AWQ checkpoints in vLLM without FP16 staging OOM:
vllm serve unsloth/Llama-3.1-8B-Instruct-bnb-4bit \
  --quantization bitsandbytes \
  --load-format bitsandbytes \
  --max-model-len 8192

Frequently Asked Questions