HashiConf The full conference agenda's now LIVE. Buy your pass and plan your schedule. Register
Well-Architected Framework
Well-Architected Framework Home

Plan for failover

Failover planning is essential for maintaining service availability when critical infrastructure components fail unexpectedly. While performance degradation planning addresses gradual issues, failover strategies handle sudden infrastructure losses that could cause complete service outages.

Effective failover planning involves implementing redundant systems, automatic health monitoring, and intelligent traffic routing that can detect failures and redirect traffic to healthy instances without manual intervention. This approach ensures your applications remain available even when individual components or entire regions experience failures.

Your failover strategy should include load balancing, service discovery, and multi-region deployment capabilities that work together to provide seamless failover experiences for your users.

Implement load balancing with health checks

Load balancers provide stability for your applications in two critical ways. First, they distribute traffic across multiple application instances, reducing the load on any single instance and preventing overload scenarios. Second, load balancers with built-in health checks can automatically detect when instances fail and stop sending traffic to unhealthy instances.

Configure load balancers across all major cloud providers using Terraform to ensure consistent deployment and management. Use the aws_lb resource for AWS, the azurerm_lb resource for Azure, and the google_compute_forwarding_rule for GCP.

Set up health check endpoints on your applications that load balancers can monitor to determine instance health. These endpoints should check critical application dependencies and return appropriate status codes that indicate whether the instance is ready to receive traffic.

Configure load balancer health check settings to match your application's characteristics, including appropriate timeout values, check intervals, and failure thresholds that balance responsiveness with stability.

Configure service discovery and mesh routing

Implement service discovery and service mesh solutions like Consul to ensure your applications only send requests to healthy instances without needing to route all the way up to the load balancer and back down. This approach provides more granular control over traffic routing and enables faster failover responses.

Use Consul's internal DNS capabilities to automatically route requests to healthy service instances. This DNS-based approach allows applications to discover and connect to services without hardcoded endpoints, enabling dynamic failover as services become available or unavailable.

Configure Consul to automatically route requests to services as they dynamically spin up and shut down, or otherwise become unresponsive. This automatic routing ensures that your applications can adapt to changing infrastructure conditions without manual intervention.

Implement circuit breaking patterns in your service mesh to prevent cascading failures when downstream services become unhealthy. Circuit breakers can temporarily stop requests to failing services, allowing them to recover while maintaining overall system stability.

Deploy multi-region and multi-cloud infrastructure

Create multi-region and multi-cloud deployments with Terraform to ensure your infrastructure can survive regional outages or cloud provider issues. This approach provides geographic redundancy and reduces dependency on single points of failure.

Use Terraform to reuse your configuration across multiple regions and cloud providers, ensuring consistent infrastructure deployment regardless of the target environment. This consistency reduces configuration drift and makes failover procedures more reliable.

Implement Terraform Stacks to roll out the same configuration from testing environments to staging and production, ensuring that your infrastructure changes are stable and production-ready before deployment. This approach reduces the risk of configuration-related failures during failover scenarios.

Configure your applications and services to be aware of multi-region deployments, including proper data replication, session management, and cross-region communication patterns that support seamless failover between regions.

Next steps

In this section of Design resilient systems, you learned about implementing failover strategies, including load balancing with health checks, service discovery and mesh routing, and multi-region infrastructure deployment. Plan for failover is part of the Design resilient systems pillar.

Refer to the following documents to learn more about failover and resilient systems:

If you are interested in learning more about failover and high availability, you can check out the following resources:

Edit this page on GitHub