Routing

Every model in the catalog can be served by one or more routes (independent upstreams). For each request the routing engine builds the list of eligible routes, ranks them with a strategy, then executes with retry and fallback down that list. You never address a route directly — you ask for a model and, optionally, express a preference.

How candidates are chosen

Providers are filtered before ranking. A provider is eligible for a request when it:

is enabled and serves the requested model (with a provider-specific model mapping),
supports the required capabilities (e.g. streaming, tools, vision, json_mode) for this request,
is not excluded by your organization's routing policy or the request's routing.providers,
has a closed or half-open circuit — providers whose circuit breaker is open after consecutive failures are skipped until a probe succeeds.

If no route survives filtering the request fails with 503 no_provider_available. Each rejected candidate keeps its reason; the dashboard's request detail shows the full candidate list as anonymous routes.

Strategies

Strategy	Ranking	Use when
`priority_cheapest` (default)	Provider priority (asc), then estimated cost (asc), then observed latency (asc)	You want the operator's preferred providers, cheapest first among equals.
`cheapest`	Estimated cost (asc), then priority	Cost matters most; latency and preference are secondary.
`fastest`	Exponentially-weighted average latency (asc; unknown latency sorts last), then cost	Interactive UIs where time-to-first-token dominates.
`priority`	Priority (asc), then the policy's `provider_order`, then cost	Strict operator-defined ordering.
`highest_availability`	Recent success rate (desc), then latency, then cost	Batch or critical paths where completion matters more than price.
`smart`	Not yet available. Accepted for forward compatibility; the engine falls back to `priority_cheapest` and records reason `smart_fallback`.	—

Cost used for ranking is the estimated cost of this request (see Pricing) using each candidate's price sheet. Latency and success rate come from a per-provider health record updated after every attempt.

Per-request override

Add the routing object to any inference request (chat, completions, responses, embeddings, images, speech, video, music). It is validated, applied, then stripped before the request reaches a provider.

json

{
  "model": "meta/llama-3.1-70b-instruct",
  "messages": [{ "role": "user", "content": "Summarise this…" }],
  "routing": {
    "strategy": "fastest",
    "providers": ["groq", "fireworks"]
  }
}

Field	Effect
`strategy`	Replaces the organization's default strategy for this request only.
`providers`	Whitelist of provider slugs (max 10). Only these are eligible; order is used as a tie-breaker for `priority`. Slugs that do not serve the model are ignored.

Overrides cannot bypass disabled providers, open circuits, or your key's allowed_models scope.

Organization policy

Your organization's default strategy, provider ordering, exclusions, and the max_retries / max_fallbacks budgets are configured in the dashboard under Routing. Per-request overrides take precedence over the organization policy, which in turn takes precedence over the platform default.

How the decision is exposed

Responses never name the upstream that served them. The identity of a provider is platform information, not customer information — API responses carry X-Request-Id, X-LLM-Model and X-LLM-Cost-Micro only, and there is no provider field on any response body or stream chunk.

Response headers

Header	Content
`X-Request-Id`	Unique id for this request. Use it to look the request up in the dashboard.
`X-LLM-Model`	Resolved gateway model id.
`X-LLM-Cost-Micro`	Charged amount in micro-USD (the final usage chunk is authoritative on streams).

Dashboard

Routing is visible in the dashboard, with upstreams anonymised as routes. For each model the routes are numbered Route 1, Route 2, … — the numbering is stable for that model, so Route 2 means the same thing on the Routing page, in a request's candidate list and in the request detail.

The Requests explorer shows, per request, every candidate route with its eligibility result, the ranking inputs (estimated cost, recent latency, success rate), each attempt's outcome and latency, and the route that finally answered.

Health and circuit breaking

Each provider attempt records success, error, timeout or 429 with its latency. After a configurable number of consecutive failures a provider's circuit opens and it is skipped for a cool-down period; the gateway then admits a few probe requests (half-open) and closes the circuit on success. 429s count as half a failure. Background probes every 60 seconds can close an open circuit early without spending your credits. The derived state (healthy, degraded, unhealthy, disabled) is visible per route and model on the dashboard's Routing page.