Mistral Small 24B Instruct (2409) VRAM Requirements Guide
High-efficiency reasoning & math model. Perfect fit for 16GB VRAM (RTX 4060 Ti / 4070 Ti Super) at Q4_K_M.
>98.2% retention. 14.2 GB weights, leaving +6.2 GB headroom on Dual NVIDIA RTX 3060 Rig (24GB VRAM).
Near-lossless 8-bit precision requiring 29.9 GB VRAM for benchmark evaluations.
Uncompressed FP16 weights taking 52.6 GB. Demands multi-GPU or datacenter accelerators.
Architectural Memory Breakdown & Attention Scaling
Mistral Small 24B is a dense 24-billion parameter model featuring 56 layers with 32 query heads and 8 GQA key-value heads (`headDim = 128`). Requiring 1.75 GiB of FP16 KV-cache at 8k context and 14.2 GB of Q4_K_M weights, total memory reaches 16.8 GB. It sits directly on the boundary between 16GB and 24GB GPUs, offering exceptional code and reasoning density.
On 16GB graphics cards, switch quantization to `IQ4_XS` (13.4 GB weights) and set `OLLAMA_KV_CACHE_TYPE=fp8`. This keeps total memory under 15.2 GB, preventing OOM crashes on the RTX 4060 Ti 16GB and RTX 4070 Ti Super.
Dual NVIDIA RTX 3060 Rig (24GB VRAM) provides 24.0 GB physical VRAM (+6.2 GB operational margin at 8k context).
Run Mistral Small 24B Instruct (2409) Locally
ollama run mistral-small-24b:q4_K_Mvllm serve Mistral AI/mistral-small-24b --max-model-len 8192 --gpu-memory-utilization 0.95Frequently Asked Questions
Can Mistral Small 24B run on a 16GB graphics card?
At standard Q4_K_M (16.8 GB), it slightly exceeds 16GB. However, using IQ4_XS or Q3_K_M brings it under 15.2 GB, fitting safely inside 16GB VRAM.
What hardware provides the best uncompressed experience for Mistral Small 24B?
A 24GB GPU like the RTX 3090 or RTX 4090 runs Mistral Small 24B at Q5_K_M or Q8_0 with massive 32k context and zero compromises.
Test Custom Hardware & Context Windows
Simulate Mistral Small 24B Instruct (2409) across all 31 GPUs, adjust context up to 131,072 tokens, or calculate RunPod break-even costs.
Open Mistral Small 24B Instruct (2409) in GPU Garden Matchmaker →