OpenAI Compatible Endpoints
One request shape became the common language of LLM serving. Learn it once and the same client talks to a hosted API or a model running on your own machine.
One Client, Many Backends
Compatibility means changing a URL, not rewriting an integration
The Only Line That Changes
Same SDK, same request body, same response parsing
Your application
Hosted API
https://api.provider.com/v1
llama.cpp
http://localhost:8080/v1
Ollama
http://localhost:11434/v1
vLLM
http://localhost:8000/v1
This is why the inference engines nearly all ship an OpenAI-compatible server: it makes them drop-in replacements for each other, and for hosted providers, without touching client code.
Anatomy of a Chat Completion
A list of messages in, one message out
{
"model": "my-model",
"messages": [
{ "role": "system",
"content": "You are terse." },
{ "role": "user",
"content": "Define a token." }
],
"temperature": 0.7,
"max_tokens": 200,
"stream": false
}
{
"id": "chatcmpl-...",
"object": "chat.completion",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "A chunk of text..."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 23,
"completion_tokens": 58,
"total_tokens": 81
}
}
Two fields deserve attention. finish_reason tells you why generation stopped — stop for a natural ending, length when it hit the limit and was cut mid-thought. usage is the billing and budget record, counted in tokens, not characters.
Message Roles
The conversation is an array you resend in full on every request
| Role | What it carries | Notes |
|---|---|---|
| system | Standing instructions: persona, rules, output format. | Conventionally first, and usually weighted more heavily than user turns. |
| user | What the person said, plus any documents pasted in. | Untrusted content lands here — instructions inside it are not your instructions. |
| assistant | What the model said on previous turns. | You send its own history back to it; the server keeps no state between calls. |
| tool | The result of a function the model asked to run. | Paired to a preceding tool call by id — the wire form of the agent loop. |
Parameters You Will Actually Set
The sampling knobs, as they appear in the request body
| Field | Effect | Typical use |
|---|---|---|
| temperature | Flattens or sharpens the probability distribution before sampling. | Low for extraction and code, higher for open-ended writing. |
| top_p | Samples only from the smallest set of tokens whose probabilities sum to p. | Usually tuned instead of temperature, not alongside it. |
| max_tokens | Hard cap on generated length. Does not make the model concise — it truncates. | Guard against runaway output and cost. |
| stop | Strings that end generation as soon as they appear. | Cutting cleanly at a delimiter in structured output. |
| stream | Return incremental chunks instead of waiting for the whole reply. | Anything a person watches happen. |
| seed | Requests reproducible sampling where the backend supports it. | Testing and evaluation. Best-effort, never guaranteed. |
What these do to the output is covered in depth on the parameters page.
Streaming
Set stream: true and the response becomes a sequence of server-sent events
data: {"choices":[{"delta":{"role":"assistant"},"index":0}]} data: {"choices":[{"delta":{"content":"A"},"index":0}]} data: {"choices":[{"delta":{"content":" chunk"},"index":0}]} data: {"choices":[{"delta":{"content":" of"},"index":0}]} data: {"choices":[{"delta":{},"index":0,"finish_reason":"stop"}]} data: [DONE]
delta, not message
Each chunk carries only the new fragment. The client concatenates them; nothing resends the full text so far.
[DONE] ends it
A literal sentinel, not JSON. Parsing it as JSON is the classic first bug when writing a client by hand.
Chunks are not tokens
A chunk usually holds one token, but that is a convention rather than a promise. Never count tokens by counting chunks.
Usage may be missing
Many backends omit the usage block when streaming unless you ask for it, so cost accounting needs its own path.
Embeddings
The other endpoint worth knowing — text in, vector out
{
"model": "my-embedding-model",
"input": [
"A token is a chunk of text.",
"Tokens are how models read."
]
}
{
"object": "list",
"data": [
{ "index": 0,
"embedding": [0.021, -0.44, ...] },
{ "index": 1,
"embedding": [0.019, -0.41, ...] }
],
"usage": { "total_tokens": 14 }
}
No text is generated. Each input becomes a fixed-length vector whose direction encodes meaning, so similar sentences land near each other. This is the machinery behind semantic search and the retrieval half of an agent's long-term memory. Vectors from different models are not comparable — re-embed everything when you switch.
The Endpoint Surface
Most compatible servers implement the first four
| Endpoint | Purpose | Support |
|---|---|---|
| /v1/chat/completions | The main one. Messages in, an assistant message out, with optional tool calling. | Effectively universal |
| /v1/models | Lists what this server can serve. Handy for discovering the right model id. | Effectively universal |
| /v1/embeddings | Vectors for search and retrieval, if the server has an embedding model loaded. | Common |
| /v1/completions | The older, plain-text form with a single prompt string and no roles. | Common, legacy |
Where Compatible Stops Being Identical
The shape matches; the behaviour behind it varies
Silently ignored parameters
A backend that does not implement a field usually accepts it and moves on rather than erroring. Your carefully chosen seed or penalty may be doing nothing.
The model field varies
Hosted APIs expect a published name; local servers may want a file name, a tag, or ignore the field entirely because only one model is loaded.
Tool calling is uneven
The request shape is standard, but reliability depends on the underlying model's training. Some local models emit malformed calls or ignore the tools entirely.
Token counts are not portable
Each model family tokenizes differently, so the same text yields different usage numbers across backends. Budgets do not transfer between them.
Context limits differ sharply
A prompt that fits a hosted model may overflow a local one, where the window is often set at load time and is smaller than the model could support.
Local auth is often nominal
Local servers commonly accept any api_key, and many bind to all interfaces by default. Treat an open endpoint on your network as exactly that.