VRAM and Throughput
Two calculators over the same model: one for safetensors weights at full precision, one for the GGUF quantizations llama.cpp ships. Both answer the questions that decide whether you can run something — will it fit, and how fast will it decode — across one card or several.
The Calculator
Pick a weight format, describe the model, choose a card. Every field is editable — the presets are starting points, not gospel.
Model
Safetensors precision
GGUF quantization
Context and KV cache
Multi-token prediction
Hardware
| Weights | |
| KV cache | |
| Activations and framework | |
| Total |
How the Estimate Works
Three formulas, none of them complicated. Knowing which one binds is the useful part.
Weights
Total parameters times bytes per parameter. For GGUF that byte count is fractional, because k-quants assign different widths per tensor — attention and embedding tensors usually keep more precision than the feed-forward bulk. All parameters count here, including every expert in a mixture, because they all have to be resident.
KV cache
Two tensors per layer, one key and one value, for every token of every sequence: 2 x layers x KV heads x head dim x bytes. The same formula as the KV cache page, and at long context it can exceed the weights outright.
Decode throughput
Decoding is bandwidth-bound: every token requires reading the active weights and the cache once. Tokens per second is roughly bandwidth divided by bytes read per token, scaled by the share of peak a real engine reaches. Only active parameters are read, which is why a mixture decodes like a small model while occupying memory like a large one.
Memory always pools. Bandwidth only sometimes does.
Two cards give you twice the VRAM either way, which is what lets a model fit at all. Whether they give you twice the speed depends entirely on how the model is split, and the two answers are far apart.
Tensor parallel shards every layer across the cards, so they read their shards simultaneously and bandwidth genuinely adds — minus the per-layer collective, which is why the estimate scales at roughly 90 percent for two cards and less beyond that. On PCIe without NVLink it is worse.
A pipeline or layer split — what llama.cpp does by default when you spread layers across devices — assigns whole layers to whole cards. For one sequence they run in turn, each idle while another works, so a token still waits on the same total bytes. You get the memory and none of the speed. Concurrent requests do benefit, because different cards can then be busy on different sequences.
Where This Will Be Wrong
It is an estimate, not a benchmark. These are the places it drifts — worth reading before buying hardware on it.
| Assumption | Why it drifts |
|---|---|
| Decode is purely bandwidth-bound | True for single-stream decode. Under large batches it moves toward compute-bound, and this will under-predict aggregate throughput. |
| Weights are read once per token | Caches, fused kernels and reuse across a batch all reduce real traffic. The single-sequence case is the honest worst case. |
| Overhead is a rough allowance | Activations scale with batch and context, and every runtime has its own allocator behaviour and fragmentation. |
| Effective bits per weight | The k-quant figures are typical values. Real file size depends on vocabulary size and how large the embedding and output tensors are next to the body. |
| Uniform memory bandwidth | Unified-memory machines share bandwidth with the CPU, and partial offload to system RAM collapses throughput far below anything shown here. |
| Cards split evenly | Pooled VRAM assumes a balanced split, but whole layers cannot straddle two cards, so a real split leaves one card fuller than the others. Tensor-parallel scaling also depends on the interconnect — NVLink beats PCIe, and mixed cards run at the pace of the slowest. |
| Routing is free | Mixture routing adds traffic and can stall on imbalance, so the active-parameter figure is an upper bound. |