System Philosophy

1. Idempotency by Design

• Every background task (Celery job, webhook handler, message sender) must be idempotent.
• Multiple execution of the same event (e.g. Meta delivering a webhook twice) must produce the exact same outcome without creating duplicate records or double-sending messages.

2. Queue Isolation

• Separate high-priority short tasks (heartbeats, live chat delivery) from bulk low-priority tasks (mass broadcasts).
• Use distinct Celery queues:

  • scheduler

    : Heartbeat & stale job cleanup (guaranteed instant execution).

  • jobs

    : Broadcast batch processing.

  • celery

    : Default asynchronous tasks.

3. Graceful Degradation

• If Redis is temporarily unreachable, WebSockets should degrade gracefully without crashing the core Django REST API server.
• If OpenRouter AI services experience high latency, customer messages should be queued for human operators without blocking the messaging pipeline.

4. Stateless ASGI Servers

• Daphne ASGI processes must remain completely stateless.
• Session data, WebSockets state, and channel group memberships are stored in Redis, allowing horizontal scaling across multiple Daphne nodes.