Enter the model
Parameters (presets for popular models), layers, hidden size, heads and KV heads for the cache estimate.
Fine-tuning
Model Size & Quantization Calculator
How big is a 7B model at Q4_K_M, and does it fit on an 8 GB card with a 4k context? This calculator turns a parameter count into file size at every common precision and GGUF level, adds the KV cache for a context length, and shows the total against GPU and RAM sizes, so you know before the download.
Loading the tool…
How it works
Pick precisions
fp32, bf16/fp16, int8, int4, and GGUF Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K with their measured bits per weight.
Set context
Context length and concurrent sequences give KV cache; the table shows total memory and which devices it fits.
Formats, limits and the numbers that matter
Bits per weight
GGUF K-quants are not exactly their name: Q4_K_M is about 4.85 bits per weight, Q5_K_M about 5.7, Q8_0 8.5, because of block scales. The tool uses measured averages.
KV cache
Per token: 2 x layers x kv_heads x head_dim x bytes. Grouped-query attention (kv_heads < heads) cuts it substantially; Llama-3 8B is about 128 KB per token at fp16.
Quality
Q4_K_M is the usual sweet spot; Q3 and Q2 lose noticeably on reasoning. int8 is nearly lossless for most models.
Common questions
Does file size equal VRAM?
Roughly, plus KV cache, plus a few hundred MB of buffers; the tool adds a margin.
CPU offload?
Enter GPU layers to split the model between VRAM and RAM the way llama.cpp does.
Vision models?
The same arithmetic applies to any parameter count; the KV cache term is zero for CNNs.