Troubleshooting

​

API key returns 401

• Key not assigned to a team — every key must belong to a team. Check that the key was created with a valid team_id. In the admin UI, go to Admin → Keys and verify the team assignment.
• Key has expired — if the key was created with an expires_at date, it stops working immediately at that time. Create a new key with a future or no expiry date.
• Using the wrong Authorization format — the header must be Authorization: Bearer ml-your-key. Check for extra spaces or missing Bearer.
• Admin key vs team key — admin endpoints (/admin/...) require the admin key. Team keys only work for inference endpoints (/v1/...).

# Verify the key exists and its team assignment
curl http://localhost:8180/admin/keys \
  -H "Authorization: Bearer $ADMIN_KEY"

​

Requests are slow or timing out

• Upstream provider is slow — check the upstream provider’s status page. Latency from OpenAI, Anthropic, or other providers varies by model and time of day. Consider configuring failover routing so slow upstreams trigger automatic fallback.
• request_timeout_ms is too low — if you’ve set a custom timeout in your gateway config, increase it. Large prompts and slow models can take 30-60 seconds.
  gateway:
    request_timeout_ms: 120000  # 120 seconds
  
• Guardrail webhook latency — if you’ve configured a webhook-based guardrail, the webhook is called synchronously and adds to request latency. Check your webhook’s response time. Consider hosting the webhook closer to ManyLayers or caching results.
• Cache miss on all requests — semantic caching requires the embedding model to be available. If the embedding provider is slow or rate-limited, cache lookups add latency. Check embedding model response times.

​

Guardrail is blocking valid requests

• Regex pattern is too broad — review the regex patterns in your guardrail policy. A pattern like \d{4} will match any four-digit number, not just credit card fragments. Test patterns against real request samples before deploying.
• Policy is in block mode instead of monitor mode — if you’re still tuning a policy, set it to monitor mode first. You’ll see which requests trigger it in the audit log without blocking real traffic.
  guardrail:
    mode: monitor  # change to "block" after tuning
  
• Checking the audit log for details — the audit log records which guardrail triggered and what pattern matched. Query recent entries filtered by your team to see what’s being caught.
  curl "http://localhost:8180/admin/audit?team=my-team&status=400" \
    -H "Authorization: Bearer $ADMIN_KEY"
  

​

Knowledge base search returns poor results

• Documents haven’t finished indexing — check indexing status. Uploading many documents or large PDFs takes time. Wait for all documents to reach indexed status before evaluating quality.
  curl http://localhost:8180/admin/indexing/status \
    -H "Authorization: Bearer $ADMIN_KEY"
  
• Embedding model mismatch — if you changed the embedding model after uploading documents, old embeddings use a different vector space. Trigger a re-embed job to regenerate embeddings with the current model.
  curl -X POST http://localhost:8180/admin/kbs/$KB_ID/reembed \
    -H "Authorization: Bearer $ADMIN_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "text-embedding-3-small"}'
  
• Chunk size is suboptimal — very small chunks lose context; very large chunks dilute relevance. For factual Q&A, try 512-768 token chunks. Adjust in the knowledge base settings and re-embed.
• Reranking is not enabled — enable a reranker model on the knowledge base to improve result ordering. Reranking significantly improves quality when initial retrieval returns mixed-relevance results.

​

Gateway not starting

• Config YAML syntax error — run your YAML through a linter. A malformed value (an unquoted string that looks like a boolean or number, an unclosed brace) will prevent the config from loading.
• Database not reachable — ManyLayers requires a PostgreSQL database. Check that the database is running, the connection string is correct, and network connectivity is working.
  # Test database connectivity
  psql $DATABASE_URL -c "SELECT 1"
  
• Migration not completed — on first boot, ManyLayers runs database migrations. If these fail (usually due to insufficient permissions), the service won’t start. Check logs for migration errors and ensure your database user has CREATE TABLE and ALTER TABLE permissions.
• Port already in use — if port 8180 (or your configured port) is in use by another process, the gateway can’t bind to it. Check with lsof -i :8180 and stop the conflicting process.
• License not found (SaaS/enterprise mode) — if you’re running in a mode that requires a license, ensure the license file or environment variable is present and valid. Check logs for license error messages.

​

Still stuck?