Runbook: CloudWatch Alarm observing the AWS/ApplicationELB UnHealthyHostCount metric

Application Load Balancer (ALB) report the UnHealthyHostCount metric to CloudWatch. A unhealthy host (or target) fails the health check that you configured for your target group.

A healt check is defined as follows:

• Interval of the health check (e.g., every 15 seconds)
• Path used for the HTTP GET request send to the target (e.g., /ping)
• Expected HTTP response code from the target (e.g., 200,204)
• Timeout of the GET request (e.g., 10 seconds)

A target is unhealthy if it fails M health checks in a row.

Runbook

A step by step guide to reacting to a CloudWatch Alarm observing the AWS/ApplicationELB UnHealthyHostCount metric.

UnHealthyHostCount is high

Are you reading this runbook in Slack? If yes, proceed to step 1. If not:
a. Go to the Target Groups page of the EC2 service in the AWS Management Console
b. Select your target group
c. Proceed to step 2.

1. Follow the Details Quick Link to access the details of the target group.
2. Select the Targets tab.
3. Can you see targets with an unhealthy status? If not, proceed to step 4. If yes:
   a. Can you see an Instance ID column? If yes, proceed to step 7.
   b. Can you see an IP address column? If yes, make a note of the IP address and proceed to step 6.
4. Can you see targets with an healthy status? If not, proceed to step 5. If yes:
   a. Can you see an Instance ID column? If yes, your targets are EC2 instances. Click on one of the instances. Select the Tags tab. Can you see a aws:autoscaling:groupName tag?

   I. If yes, make a note of the value. Go to the [Auto Scaling Groups page of the EC2 service in the AWS Management Console](https://console.aws.amazon.com/ec2/autoscaling/home#AutoScalingGroups:). Search for the name you noted. Select the **Activity History** tab. In the tab, search for `Terminating EC2 instance`. Starting with the first row, expand row by row and check if the cause contains `ELB system health check failure`. If yes, proceed to step 7.
   II: If not, EC2 instaces are (de)registred in a custom way. Find out how this mechanism works and where you can find a list of previously deregistered instances. Proceed to step 7.
   

   b. Can you see an IP address column? If yes, make a note of the IP address and proceed to step 6.
5. You can not see any targets. Reload the page from time to time. If targets pop up proceed with step 3. Otherwise, targets are (de)registred in a custom way. Find out how this mechanism works. End of runbook.
6. Go to the Network Interfaces page of the EC2 service in the AWS Management Console. Search for the IP address you noted.
   a. Can you see an Instance ID in the Details section? If yes, proceed to step 7.
   b. The Instance ID is set to -? If yes, proceed to step 8.
   c. If the search does not return any results, proceed to step 9.
7. Your targets are EC2 instances. Collecting logs from EC2 instances is not standardized. Find out where logs are shipped to and search for errors. End of runbook.
8. Your targets are ECS tasks.
   a. Go to the Clusters page of the ECS service in the AWS Management Console
   b. Click on the cluster in which you suspect the target. If you don’t know which cluster to consider, you have to apply the following steps to all clusters.
   c. Select the Tasks tab.
   d. Filter by Desired task status equals Stopped
   e. For each task, click on the task and check if the Private IP euqals to the IP address you noted. If no, continue with the next task. If yes, proceed to step 8.i.
   f. If you reached the end of the task list, filter by Desired task status equals Running.
   g. For each task, click on the task and check if the Private IP euqals to the IP address you noted. If no, continue with the next task. If yes, proceed to step 8.i.
   h. If you reached the end of the task list, proceed to step 10.
   i. Select the Logs tab.
   j. Search for errors in the log. End of runbook.
9. Your targets are on-premises machines. Collecting logs from on-premises machines is not standardized. Find out where logs are shipped to and search for errors. End of runbook.
10. You are running into an unknown error. End of runbook.