Serving a Model
Everything up to here was about one request. Production is about the thousandth concurrent one — where latency splits into two numbers that pull against each other, and the cheapest configuration is rarely the fastest.
Latency Is Two Numbers
Averaging them together hides the thing your users actually feel.
| Measure | What it is | Driven by | How to improve it |
|---|---|---|---|
| Time to first token | Silence before anything appears | Prompt length, queue depth, prefill compute | Shorter prompts, prefix caching, more compute |
| Time per output token | How fast text then streams | Memory bandwidth, batch size, model size | Smaller or quantized weights, faster memory |
Throughput Is Bought With Latency
Batching is the single biggest lever in serving, and it works by making individual requests slightly worse.
Decoding one sequence leaves a GPU almost idle: the weights have to be read from memory for every token regardless of how many sequences are using them. Run sixteen sequences together and the same read serves all sixteen, so throughput climbs steeply while per-request speed drops only a little.
Continuous batching is what makes this practical. Rather than waiting for a batch to finish, the scheduler drops sequences as they complete and admits new ones into the free slots on the next step. A short request no longer waits behind a long one, which is the difference between a demo and a service.
Batch size
More concurrent sequences, better hardware utilisation, more KV cache needed. The cache is what caps it, not compute.
Prefix caching
A shared system prompt is identical across requests, so its keys and values can be computed once and reused. Frequently the cheapest win available.
Speculative decoding
A small model drafts several tokens, the large one verifies them in a single pass. Accepted drafts come free; rejected ones cost a little.
Sizing the Machine
Three things compete for the same VRAM, and only one of them is fixed.
Weights are the easy part: parameters times bytes per parameter. A 7B model is about 13 GB at FP16 and roughly 4 GB at 4-bit. What people underestimate is the cache — at long contexts and real concurrency it can exceed the weights outright, which is what the calculator on the KV cache page is for.
Plan for the load you expect rather than a single request, leave headroom for fragmentation, and remember that the moment you run out, the failure is not slowness but a refused request.
What It Costs
Self-hosting is fixed cost; an API is variable cost. Which wins depends entirely on utilisation.
A rented GPU bills by the hour whether or not anyone is using it. An API bills by the token and nothing when idle. So the comparison is not a rate against a rate — it is a break-even volume.
Take a card at roughly $1/hour, about $720/month. If a served model produces on the order of a few hundred tokens a second under healthy batching, a fully saturated month is on the order of a billion output tokens. Against API pricing in the region of $0.50 to $10 per million output tokens, the same volume costs somewhere between a few hundred and several thousand dollars.
Which means the honest summary is: self-hosting wins at high, steady utilisation and loses badly at low or spiky utilisation — and most workloads are spiky. Idle GPUs are the expense nobody budgets for.
Failing Well
Model serving fails in ways ordinary web services do not, and the defaults are wrong for it.
| Failure | What users see | Defence |
|---|---|---|
| Queue saturation | Everything slows at once, then times out | Admission control — shed load rather than queue it; return a clear busy signal |
| Runaway generation | A response that never ends | Hard token cap and stop sequences on every call |
| Cache exhaustion | Requests refused despite low GPU use | Cap context and concurrency together; evict idle sessions |
| Naive retries | A slow period becomes an outage | Backoff with jitter; never retry a streamed response mid-flight |
| Upstream outage | The feature disappears | A smaller local model behind the same OpenAI-compatible interface |