Context
For an online service, an outage affecting only some regions easily goes unnoticed from headquarters, while the affected users can no longer reach the service.
Problem
Detect outages per region quickly and restore access without manual intervention, while keeping a usable record of incidents.
Approach
- Automated monitoring of domain-name availability across different geographic regions.
- Periodic checks every 10 minutes to detect blocking and network unavailability.
- Detection of regional restrictions and automatic identification of unreachable domains.
- Automatic failover logic to alternative domains to keep the service running.
- Centralised tracking of tested domains for monitoring and incident analysis.
Stack
- Python
- Monitoring
- Scheduled jobs
- Automatic failover
- PostgreSQL
- REST APIs
Status
Client engagement
Engagement delivered through ORIA; client details not published.
What I learned
Availability has to be measured where users are, not from your own office. And automatic failover is only worth it if every decision stays logged and explainable afterwards.