Rate Limiting

​

Where limits come from

​

How it works

• Every matching ceiling must allow the request. A request under a 100/min workspace rule, a 50/min team rule and a 10/min user-on-model rule has to satisfy all three. The narrowest scope can never raise a ceiling a broader scope set.
• Outermost first. When several refuse, you are told about the outermost one — organization before workspace, team, service account, user, API key.
• Cache hits still count as requests. Request ceilings are checked before the cache lookup; token ceilings are checked after it, so a hit is not charged tokens.
• Tokens are settled after the response. Admission charges the prompt; the completion is charged once known. A long answer can leave the counter empty, and the overspend is repaid out of the next window. A single prompt larger than the whole token ceiling can never fit and is refused with tpm_exceeded.
• Windows. With Redis the window slides: the previous interval is weighted by how much of it is still inside the trailing window, so capacity returns continuously instead of resetting on the minute. The in-memory backend uses a token bucket that holds the whole ceiling as burst and refills evenly across the window.

​

Rate-limit policy structure

{
  "name": "Support team on gpt-4o",
  "scope": "workspace",            // workspace (default) | organization
  "team_id": "team_9",             // at most one of: team_id, service_account_id, user_id, api_key_id
  "model_name": "gpt-4o",          // logical model; "" = every model
  "applies_per": "user",           // "" shared | user | model | "user,model": one counter each
  "requests_per_minute": 60,
  "requests_per_hour": 0,
  "requests_per_day": 10000,
  "tokens_per_minute": 200000,
  "tokens_per_hour": 0,
  "tokens_per_day": 0,
  "enabled": true
}

​

Key fields

​

Common configurations

curl -X POST https://app.manylayers.io/admin/gateway/policies/rate-limits \
  -H "Authorization: Bearer ml_pat_..." -H "Content-Type: application/json" \
  -d '{"name": "Support burst", "team_id": "team_9",
       "requests_per_minute": 60, "requests_per_day": 10000}'

{"name": "o1 per user", "model_name": "o1", "applies_per": "user",
 "requests_per_minute": 5, "tokens_per_day": 500000}

curl -X PATCH "https://app.manylayers.io/admin/keys/key_123?workspace_id=ws_123" \
  -H "Authorization: Bearer ml_pat_..." -H "Content-Type: application/json" \
  -d '{"rpm_limit": 100, "tpm_limit": 20000, "budget_usd_monthly": 50, "budget_reset_period": "monthly"}'

teams:
  - name: engineering
    rpm_limit: 60
    tpm_limit: 100000
    models: ["gpt-4o", "gpt-4o-mini"]

​

The refusal

HTTP/1.1 429 Too Many Requests
X-ManyLayers-Limit: team
Retry-After: 60

{"error": {"message": "Team rate limit exceeded", "type": "rate_limit_error", "code": "rpm_exceeded"}}

​

Backends

​

Next steps