Our highly available and fault-tolerant web application infrastructure so far needs some final validation to ensure it works as expected. Now we’ll simulate failure cases, check recover behavior, and confirm if DNS has failed.
Step 1: Access Validation to The Application
With the new launch template, let’s re-run the new auto scaling group to ensure the same landing page is accessible via our ALB. Ensure we can access the application properly via the ALB.
In case there is any issue in accessing the application, please check the following:
- Check your Target Group health
- Ensure EC2 instances are running and in service
- Confirm your security group allows HTTP (port 80)
Step 2: Simulate EC2 Instance Failure
Next, we will test the auto scaling group ability by stopping / terminating one of the EC2 instance within ASG.
- Go to EC2 → Instances
- Select one instance in the Auto Scaling Group
- Click Instance state → Stop instance
Within several minutes, ASG will initialize a new instance to replace the stopped instance
With below final result where it shows how ASG handle the failure within to ensure the application keep running.
Step 3: Simulate ALB Failure, Observe DNS Failover
Here we are testing whether the Route 53 fails and it will detect via its health checking.
- Go to EC2 → Auto Scaling Groups
- Set desired, min desired and max desired capacity = 0 (this removes all EC2 instances)
- After around 5 minutes, the ALB health check Route 53 will show unhealthy
Then if we access our domain / ALB public DNS we should be redirected to the static failover site on S3 bucket as per discussed on the part 3.
Once tested, we can reset back the ASG instance to the desired values.
Summary
In this project we have demonstrated the AWS reliability principles such as
- Redundancy
Across AZs for all critical components like EC2 instance and EFS - Monitoring
Continuous and layered health checks and alert such as at DNS level (Route 53) and compute level (load balancer) - Self-healing
Adaptability for the service to handle the failure such as Auto Scaling Group and DNS failover (Route 53) - Isolation
Layered design that give clarity in term of boundary and responsibility
In more mission-critical services for production realm, we also can consider the following principles
- Automated backups and snapshots
Using AWS backup to automate and schedule RDS snapshots and configure EFS backup to ensure the durability as part of disaster recovery - Cross-Region Failover
Using Route 53 to do the health check on the region level and automatically deploy application in another region in order to avoid regional outage