Compute·Afterburn
Measure every endpoint. Route on the evidence.
Afterburn connects to the inference endpoints a team already runs (vLLM, SGLang, llama.cpp, Ollama or a hosted API), measures them on its own prompts, and sends each request to the one that keeps its latency limit for the least cost. Every change keeps its evidence.
- OpenAI-compatible gateway
- Benchmarks on your prompts
- Every change a revision
Inside Afterburn
A working Afterburn,
right here. Click around.
This is the console in miniature, running in your browser. Watch traffic go through a route, run a benchmark across four endpoints, decide from it, apply the change and roll it back, or race three answers in the playground.
afterburn/app/Simulated in your browser · sample dataOverview route support-chat · revision 1
Requests
0none failedFirst token, p95
–limit 300 msCost
$0.000priced endpoints onlyAnswering
4 / 4endpoints upLatest requests live
Who answered
First token, recent average
Everything in this window is simulated with sample endpoints and prompts; nothing is sent anywhere. The real console measures your own endpoints with the same screens.
The problem
Tuned once,
never measured again.
Most inference endpoints are chosen from a launch-day benchmark and kept long after the models, the engines and the traffic have moved on.
- Picked on a hunch
An endpoint is chosen once, from someone else’s numbers, on prompts that look nothing like yours.
- Latency nobody owns
The 95th percentile creeps up a few milliseconds at a time, and nobody sees it until users do.
- No baseline
When an engine, a model or a provider changes, there is nothing kept to say whether it helped.
The loop
Observe. Decide.
Route. Prove.
One loop, running all the time. Each panel is laid out the way the console shows it, filled with sample data.
Observe
Afterburn measures each endpoint the way users meet it: time to first token, time between tokens, the whole answer and its errors, timed on the stream. Where the engine publishes Prometheus metrics, it reads the queue and the KV cache too.
Endpoint · vllm-h100Sample data
- model
- Llama 3.1 8B Instruct
- first token, p95
- 45 ms
- between tokens, p50
- 11 ms
- KV cache in use
- 64%
- errors
- 0 of 40
Decide
A route has a limit: first token under a p95, the whole answer under a p95, an error budget. A benchmark keeps only the endpoints that hold it, ranks them by measured cost per million tokens, and says why the others were turned down.
Route · support-chatSample data
- limit
- first token p95 ≤ 200 ms · errors ≤ 2%
- vllm-h100
- 45 ms · $0.24 / 1M · selected
- llama-cpp-mac
- 126 ms · qualifies
- sglang-l40s
- 254 ms · over the limit
Route
Apps call one OpenAI-compatible URL and name a route as the model. Each request goes to the route's primary endpoint and falls through to the next on errors and timeouts. Every change is a revision; going back is one step.
Gateway · support-chatSample data
- primary
- vllm-h100
- fallbacks
- llama-cpp-mac, sglang-l40s
- revision
- 2, from 1
- rollback
- one step
Prove
Every request through the gateway is logged with its endpoint, latency, tokens and cost. When a route changes, Afterburn holds the hour before against the hour after, so each decision keeps its own receipt, good news or not.
Change · revision 1 → 2Sample data
- first token, p95
- 254 → 45 ms
- total, p95
- 256 → 47 ms
- cost per 1M output
- $0.16 → $0.24
- requests compared
- 36 before, 44 after
Benchmarks
Where each endpoint
breaks the limit.
A benchmark is the same prompts, at a chosen concurrency, to every endpoint. Run it at several loads and you see where each one stops keeping the route's limit; set it against price and you see which one should serve it. Switch the prompt set: long contexts change the answer.
First token p95 as load riseslog scale · sample data
vllm-h100sglang-l40sllama-cpp-macopenrouter-8b
Cost against speed, 8 at onceSample data
sglang-l40s keeps the limit for the least: $0.177 per 1M output tokens.llama-cpp-mac is unpriced (local) and not on the chart.
The gateway
One URL for every app.
The route decides the rest.
Point any OpenAI client at Afterburn and name a route instead of a model. When the primary fails, the next endpoint takes the request before anything has been sent back, and the response says which endpoint answered.
Route support-chat with its share of requests by endpoint. Sample data.
Your stack
It sits in front.
It replaces nothing.
Anything that speaks the OpenAI-compatible API can be measured and routed: the engines a team serves itself, the models on a laptop, and hosted providers. Keys are sealed at rest, and every change is on the audit trail.
- vLLMServing engine
- SGLangServing engine
- TensorRT-LLMServing engine
- llama.cppLocal models
- OllamaLocal models
- LM StudioLocal models
- OpenRouterHosted models
- PrometheusEngine metrics