Routing

​

Endpoint routing

router:
  strategy: least_inflight   # least_inflight | round_robin | prefix_affinity | score
  health_interval: 15s       # how often ejected endpoints are probed for recovery
  failure_threshold: 3       # consecutive failures before ejecting an endpoint
  cooldown: 30s              # base ejection window
  session_ttl: 10m           # sticky session lifetime
  prefix_bytes: 1024         # prompt prefix bytes for prefix_affinity
  health_threshold: 0        # eject when the success-rate average falls below this (0 = off, max 1)
  restore_health: 0.5        # health an ejected endpoint returns with (0 = default 0.5)
  max_ejection_duration: 0   # cap on the escalating ejection window (0 = uncapped)
  respect_retry_after: true  # use a rate-limited provider's Retry-After as the ejection window

​

Sticky sessions

​

Upstream health

• Ejection. An endpoint is taken out of rotation after failure_threshold consecutive failures, or when its health average falls below health_threshold. Connection errors, timeouts, 5xx, 429, 408 and 401/402/403 from the provider count as failures; other 4xx responses reject what the caller sent and do not.
• Window. The ejection lasts cooldown, multiplied by the number of ejections since the last success, up to max_ejection_duration. When the provider states its own wait (Retry-After) and respect_retry_after is on, that wait is used instead, unescalated.
• Recovery. Every health_interval the gateway probes ejected endpoints with GET /v1/models. One that answers comes back at restore_health, so it earns traffic back instead of taking its old share at once. A real success clears the escalation.
• Failover. With several endpoints, a request that fails before any byte reaches the client moves to the next endpoint. If every endpoint is ejected, the request fails immediately with provider_unavailable.

curl https://app.manylayers.io/admin/gateway/health \
  -H "Authorization: Bearer ml_pat_..."

​

Routing configs

curl -X POST https://app.manylayers.io/admin/gateway/configs \
  -H "Authorization: Bearer ml_pat_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "resilient",
    "strategy": "fallback",
    "targets": [
      {"model": "gpt-4o"},
      {"model": "claude-sonnet"}
    ],
    "retry": {
      "attempts": 1,
      "backoff_ms": 200,
      "on_status_codes": [429, 500]
    }
  }'

​

Routing strategies

{
  "strategy": "fallback",
  "targets": [
    {"model": "gpt-4o"},
    {"model": "claude-sonnet"}
  ],
  "retry": {"attempts": 1, "backoff_ms": 200, "on_status_codes": [429, 500]}
}

{
  "strategy": "canary",
  "targets": [
    {"model": "gpt-4o", "weight": 90},
    {"model": "gpt-4o-new", "weight": 10}
  ]
}

{
  "strategy": "conditional",
  "targets": [
    {"model": "gpt-4o", "when": {"header": "X-Priority", "equals": "high"}},
    {"model": "gpt-4o-eu", "when": {"metadata": {"region": "eu"}}},
    {"model": "gpt-4o-mini"}
  ]
}

{
  "strategy": "latency",
  "targets": [{"model": "gpt-4o-mini"}, {"model": "gemini-flash"}]
}

{
  "name": "smart-chat",
  "strategy": "complexity",
  "classification_strategy": {"type": "heuristic"},
  "targets": [
    {"model": "gpt-4o-mini", "tier": "simple"},
    {"model": "gpt-4o", "tier": "medium"},
    {"model": "claude-sonnet", "tier": "complex"}
  ]
}

{
  "strategy": "single",
  "targets": [{"model": "gpt-4o"}]
}

​

Using a routing config

curl https://app.manylayers.io/v1/chat/completions \
  -H "Authorization: Bearer $MANYLAYERS_API_KEY" \
  -H "X-ManyLayers-Config: resilient" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hi"}]}'

​

Config management API