How to monitor Amazon Redshift?

Andreas Wittig – 18 Sep 2020

A data warehouse is used for analyzing GBs or TBs of data and is a core component of business intelligence. AWS offers a popular data warehouse called Amazon Redshift. AWS is responsible for most aspects of operating the data warehouse. However, there are a few metrics that you should monitor to ensure smooth operation.

How to monitor Amazon Redshift?

This article describes how to monitor a Redshift cluster. As most AWS services do, Redshift reports metrics to CloudWatch. Use those metrics to define alarms and get notified about issues via email, Slack, or Microsoft Teams.

Monitor Redshift with CloudWatch

Health

Most important, you should monitor the health of your Redshift cluster. Even though AWS is responsible for operating the data warehouse, you should know whenever there is an issue with your cluster. It might be necessary to contact AWS support or to restore your cluster from a snapshot.

The HealthStatus metric indicates whether your cluster is healthy. If so, the Redshift cluster reports a value of 1 every minute. Therefore, create a CloudWatch alarm that checks for a value of less than one.

From time to time, AWS will schedule a maintenance window for your Redshift cluster. The CloudWatch metric MaintenanceMode indicates when your cluster is unavailable because of maintenance work. Make sure to define a CloudWatch alarm checking for a value greater zero.

Quotas

The storage capacity of your Redshift cluster is limited. The smallest node types come with 160 GB storage. The biggest node type offers up to 64 TB. In both cases, you should monitor the PercentageDiskSpaceUsed metric closely. Doing so avoids that your cluster will run out of storage capacity.

As an administrator, you might want to define storage quotas for specific schemas of the data warehouse. Doing so allows you to isolate and limit different workloads running on the same Redshift cluster. The QUOTA parameter does the trick1. When defining storage quotas for schemas, you should also monitor whether a schema exceeds the quota. Therefore, I recommend creating a CloudWatch alarm verifying that the NumExceededSchemaQuotas metric does not exceed zero.

Performance

Monitoring the performance of a Redshift cluster is tricky. However, there are a few metrics that are easy to observe:

  • CPUUtilization which indicates the utilization of the CPU
  • ReadThroughput which indicates the read throughput to the storage system
  • WriteThroughput which indicates the write throughput to the storage system

Of course, it is totally fine for the cluster to work at full capacity. But keep in mind that it might help to increase the node type when your data warehouse is under high load most of the time.

Costs

When spinning up a Redshift cluster, most of the monthly costs are static. But, there are a few exceptions to that rule: concurrency scaling and managed storage.

Concurrency Scaling2 is a feature that allows your Redshift cluster to scale horizontally to process requests in parallel. The feature is free for up to 60 minutes. After that, you start paying. Costs can be significant. Therefore I recommend monitoring the ConcurrencyScalingSeconds metric. It would be best if you were notified whenever the sum within 24 hours is greater than 3600.

When using nodes of type RA3, the managed storage is billed per GB up to 64 TB. The only way to monitor storage consumption is the PercentageDiskSpaceUsed metric.

Monitoring Jump Start

In summary, I recommend to monitor the following CloudWatch metrics closely:

  • HealthStatus indicates the health of your Redshift cluster.
  • MaintenanceMode notifies about ongoing maintenance.
  • PercentageDiskSpaceUsed to monitor the remaining disk space.
  • NumExceededSchemaQuotas check for exceeded schema quotas.
  • CPUUtilization indicates the CPU utilization.
  • ReadThroughput and WriteThroughput shows the I/O throughput to the storage system.
  • ConcurrencyScalingSeconds to avoid unwanted costs due to concurrency scaling.

Therefore, you need to create and configure seven CloudWatch alarms per Redshift cluster. Want a little help with that? Check out our monitoring jump start for CloudFront.

  1. Install marbot (Slack or Microsoft Teams).
  2. Invite marbot in one of your channels.
  3. Send @marbot monitor redshift to the channel.
  4. marbot will help you to configure CloudWatch alarms with the help of CloudFormation.
  5. When creating the CloudFormation stack …
    1. Set the ClusterIdentifier parameter to the ID of your Redshift cluster.
    2. Set the NodeType parameter to the type of your cluster nodes.
    3. Review the threshold parameters.
    4. Save by clicking the Create button.
  6. marbot will notify you about issues with your CloudFront distribution.
Andreas Wittig

Andreas Wittig

Consultant focusing on Amazon Web Services (AWS). Entrepreneur building marbot.io. Author of Amazon Web Services in Action, Rapid Docker on AWS, and cloudonaut.io.

You can contact me via Email, Twitter, and LinkedIn.

Published on

marbot teaser

Chatbot for AWS Monitoring

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

Slack
Add to Slack
Microsoft Teams
Add to Teams