ManyLayers uses a message queue for all background work: connector syncs, workflow dispatch, batch polling, re-embedding jobs, training jobs, and billing rollups.
Queue backends
DB backend (default)
Kafka backend
The DB backend uses Postgres-based claim polling. It requires no additional infrastructure beyond your existing Postgres database.Best for: Single-replica on-prem deployments and teams getting started. queue:
backend: kafka
kafka:
brokers:
- kafka:9092
prefix: "" # optional topic prefix, e.g. "prod."
<topic>.dlq).Best for: Multi-replica deployments and high-throughput production environments.Starting Kafka with Docker Compose
infra/docker/docker-compose.dev.yml includes a single-node Kafka instance in the optional profile. Name the service to start it on its own:
docker compose --env-file infra/docker/.env.dev -f infra/docker/docker-compose.dev.yml up -d kafka
kafka:9092 from other stack containers, and on localhost:9192 from the host (override with KAFKA_HOST_PORT).
Then update your gateway config:
queue:
backend: kafka
kafka:
brokers:
- kafka:9092
The queue section is restart-only. Switching between db and kafka requires a gateway restart.
Message delivery guarantee
Both backends deliver messages at-least-once. Each handler re-checks the state of the underlying work before executing, so duplicate deliveries are safe — the second execution detects that the work is already done and exits cleanly. 
