How We Achieved 99.9% Uptime for 15,000 Conversational Apps
Running thousands of real-time conversational applications requires an architecture that's resilient, scalable, and fault-tolerant. Here's how we built infrastructure that keeps 15,000 apps running with 99.9% uptime.
The Challenge
When you're running conversational AI at scale, a single point of failure can take down thousands of customer-facing applications. Users expect instant responses, and every second of downtime means lost conversations, frustrated customers, and damaged trust.
Our platform needed to handle:
- 15,000+ concurrent applications each with different configurations
- Millions of messages per day with sub-second response times
- WebSocket connections that need to stay alive for hours
- Streaming responses from LLMs that can take 10-30 seconds
- Zero tolerance for data loss in conversations
- Multiple availability zones - applications run across 3+ AZs
- Load balancing - traffic distributed across healthy instances only
- Database replication - real-time replication with automatic failover
- Message queues - persistent queues survive service restarts
- Normal: Responses in 2 seconds
- Degraded: Queue requests, return in 10 seconds
- Failure: Return cached response or helpful error
- Normal: Read/write to primary
- Degraded: Read from replica, queue writes
- Failure: Serve from cache, log for retry
Architecture Principles
1. No Single Points of Failure
Every component in our stack is redundant:
If any single server, database, or even an entire data center goes down, traffic automatically routes to healthy infrastructure.
2. Graceful Degradation
When things go wrong (and they always do), the system degrades gracefully:
Example: LLM API is slow
Example: Database connection issues
We never return a 500 error when we can return a slower-but-working response.
3. Circuit Breakers
We use circuit breakers for all external dependencies:
1. Service starts failing (3 failures in 10 seconds)
2. Circuit breaker opens → stop sending traffic
3. Give service time to recover (30 seconds)
4. Test with single request (half-open state)
5. If successful → close circuit, resume normal traffic
6. If failed → stay open, try again later
This prevents cascading failures where one slow service takes down everything else.
Infrastructure Stack
WebSocket Management
Real-time conversations require persistent WebSocket connections. We use:
Key insight: Don't try to keep connections in memory. Store connection state in DynamoDB so any Lambda can handle any message.
Message Processing Pipeline
User Message
↓
API Gateway WebSocket
↓
Lambda: Route Message
↓
SQS Queue (persistent)
↓
Lambda: Process Message
↓
Call LLM API (with retry logic)
↓
Lambda: Send Response
↓
API Gateway → User
Every step is fault-tolerant:
Database Strategy
We use DynamoDB for its proven reliability:
For analytics, we stream data to S3 via Kinesis Firehose, then query with Athena.
LLM API Resilience
LLM APIs are external dependencies we don't control. Our defense:
1. Multiple providers
2. Request-level timeouts
First attempt: 10 second timeout
Retry 1: 15 second timeout
Retry 2: 20 second timeout
Fallback provider: 10 second timeout
3. Response streaming
4. Caching
Monitoring and Alerting
You can't fix what you can't see. We monitor:
Application Metrics
Infrastructure Metrics
Business Metrics
Alerting Strategy
Deployment Strategy
Zero-downtime deployments are critical when you're running 24/7.
Blue-Green Deployments
1. Deploy new version to "green" environment
2. Run smoke tests on green
3. Route 5% of traffic to green
4. Monitor for 15 minutes
5. If healthy → route 50% of traffic
6. Monitor for 15 minutes
7. If healthy → route 100% of traffic
8. Keep blue running for 1 hour (quick rollback)
9. Terminate blue environment
If anything goes wrong, we can route 100% of traffic back to blue in seconds.
Database Migrations
Schema changes are the hardest part of zero-downtime deployments.
Our approach:
1. Make changes backward compatible
- Add new column (old code ignores it)
- Deploy new code (reads new and old columns)
- Backfill data
- Deploy code that only uses new column
- Remove old column
2. Use feature flags
- New features start disabled
- Enable for internal users first
- Gradual rollout to 1%, 10%, 50%, 100%
- Instant rollback by disabling flag
Incident Response
Despite all precautions, incidents happen. Our process:
Detection (Target: <2 minutes)
Response (Target: <5 minutes)
1. Acknowledge alert - someone takes ownership
2. Assess severity - is this impacting customers?
3. Communication - notify team, update status page
4. Immediate mitigation - rollback, failover, or scale up
Resolution (Target: <30 minutes)
1. Identify root cause - logs, metrics, traces
2. Implement fix - deploy or configuration change
3. Verify fix - metrics return to normal
4. All-clear - update status page, notify team
Post-Mortem (Within 72 hours)
1. What happened - timeline of events
2. Why it happened - root cause analysis
3. What we're doing - action items to prevent recurrence
4. Share publicly - transparency builds trust
Real Incident: The API Gateway Timeout
What happened: In July 2024, we saw a spike in WebSocket connection failures.
Timeline:
Root cause: We hit the default API Gateway concurrent connection limit (10,000). During peak hours, we had 12,000+ active connections.
Permanent fixes:
1. Increased API Gateway limits to 100,000
2. Added alerting when approaching 80% of any limit
3. Implemented automatic scaling triggers
4. Set up load testing to catch limits before production
Result: Similar issues haven't occurred since. When we approached 80,000 connections in September, alerts fired and we proactively increased limits.
Lessons Learned
1. Design for Failure
Every external dependency will fail. Design your system to handle it gracefully.
2. Observability is Not Optional
You can't operate what you can't see. Invest in metrics, logs, and tracing from day one.
3. Test Failure Modes
Run chaos engineering experiments. Kill random services. Overload your APIs. See what breaks.
4. Automate Everything
Manual processes fail at 3 AM. Automate detection, mitigation, and recovery.
5. Document Runbooks
When an incident happens, you don't want engineers figuring things out. Have runbooks ready.
Our Current Stats
As of September 2024:
What's Next
We're working on:
Want to build on our platform? Get Started →
---
*Questions about our architecture? Email us at hello@ilq.ai*