Canary routing lets you split traffic between models by percentage weight, making it safe to test a new model version in production before fully committing to it.
1

Configure both models

Your gateway.yaml needs entries for both the current and the new model version:
models:
  - logical_name: gpt-4o
    provider: openai
    upstream_model: gpt-4o
    upstream_api_key: ${OPENAI_API_KEY}
  - logical_name: gpt-4o-new
    provider: openai
    upstream_model: gpt-4o-2025-06-01
    upstream_api_key: ${OPENAI_API_KEY}
2

Create a canary routing config

curl -X POST http://localhost:8180/admin/gateway/configs \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gpt4o-canary",
    "strategy": "canary",
    "targets": [
      {"model": "gpt-4o", "weight": 90},
      {"model": "gpt-4o-new", "weight": 10}
    ]
  }'
3

Test the canary config

curl http://localhost:8180/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-ManyLayers-Config: gpt4o-canary" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
Approximately 10% of requests will go to gpt-4o-new. Monitor your usage analytics to compare error rates and latency.
4

Increase the canary weight

As you gain confidence in the new model, increase its share:
curl -X PUT http://localhost:8180/admin/gateway/configs/$CONFIG_ID \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gpt4o-canary",
    "strategy": "canary",
    "targets": [
      {"model": "gpt-4o", "weight": 50},
      {"model": "gpt-4o-new", "weight": 50}
    ]
  }'
5

Complete the rollout

When satisfied, switch entirely to the new model:
curl -X PUT http://localhost:8180/admin/gateway/configs/$CONFIG_ID \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gpt4o-canary",
    "strategy": "single",
    "targets": [{"model": "gpt-4o-new"}]
  }'

​
Apply the config automatically for a team

Instead of requiring clients to send the X-ManyLayers-Config header, set the config as a team’s default: 
curl -X POST http://localhost:8180/admin/teams \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "engineering",
    "default_config_id": "'$CONFIG_ID'",
    "models": ["*"]
  }'
All requests from the engineering team will use the canary config automatically.

​
Add failover for resilience

Combine canary routing with automatic retry for extra safety: 
{
  "strategy": "canary",
  "targets": [
    {"model": "gpt-4o", "weight": 90},
    {"model": "gpt-4o-new", "weight": 10}
  ],
  "retry": {
    "attempts": 1,
    "backoff_ms": 200,
    "on_status_codes": [429, 500, 502, 503]
  }
}