Runbook: CloudWatch Alarm observing the AWS/Lambda Errors metric

Lambda reports the Errors metric to CloudWatch. The metric counts the number of errors during Lambda function executions.

Errors can be caused by:

• Execution timeout: Executions run longer than the configured timeout
• Out of memory: Executions need more memory than configured
• Handled errors: Errors the code reports back to Lambda on purpose
• Unhandled errors: Such as runtime exceptions or other errors in the code

Runbook

A step by step guide to reacting to a CloudWatch Alarm observing the AWS/Lambda Errors metric.

Errors greater than 0

Are you reading this runbook in Slack? If yes, proceed to step 1. If not:
a. Go to the Logs page of the CloudWatch service in the AWS Management Console
b. Click on the log group /aws/lambda/<Your Lambda Function Name>
c. Hit the blue Search Log Group button
d. Proceed to step 2.

1. Follow the Logs Quick Link to access the logs of the errored Lambda function.
2. Search for "Task timed out after". If no results appear, proceed to step 3. Otherwise, your function executed longer than the configured timeout. You might want to increase the function timeout (up to 15 minutes). End of runbook.
3. Search for "Process exited before completing request". If no results appear, proceed to step 4. Otherwise:
   a. Click on the stream link on the right to a result.
   b. Check the line before “Process exited before completing request” says “Duration: xxx ms Billed Duration: xxx ms Memory Size: xxx MB Max Memory Used: xxx MB”.
   c. Does Memory Size equal Max Memory Used? If yes, your function consumed more memory than the maximum you configured. You might want to increase the function memory (up to 3008 MB). End of runbook.
4. Search for ?error ?exception ?failure. If no results appear, proceed to step 5. Otherwise, there is probably an error in your code. End of runbook.
5. Clear the search query. Can you see a stack trace in your log by scrolling trough the logs? If not, proceed to step 6. If yes, there is probably an error in your code. End of runbook.
6. You are running into an unknown error. End of runbook.