Alert escalation chain

marbot tries to achieve two goals: no alert should go unnoticed and minimize interruptions. To achieve that, marbot offers several techniques for delivering alerts to a team.

Notifications vs. Alerts

marbot categorizes incoming events from your AWS infrastructure into two categories: notifications and alerts. Notifications contain informations about events that are nice-to-know (e.g., that a backup job completed successfully). Alerts are critical and require human interaction.

Acknowledge and Close

marbot expects, that team members acknowlede alerts when starting to investigate and solve the underlying issue. Press the Acknowledge button shown at the bottom of the alert to do so.

Here is what an alert looks like in Slack.

And here is the same alert in Microsoft Teams.

Here is what happens when acknowledging an alert in a channel.

1. Send direct message containing alert to user.
2. Mark the alert deliverd to the channel as acknowledged.

It is important to know, that marbot expects team members to close alerts after investigating and solving the underlying issue. To do so, press the Close button.

By the way, send the following direct message to @marbot to list all open alerts that you have acknowledged.

open alerts

In case you want to get a list of all alerts that have not yet been acknowledged, send the following direct message to @marbot.

missed alerts

Auto-close

A CloudWatch Alarm can be in one of three states:

• OK
• ALARM
• INSUFFICIENT_DATA

If a CloudWatch Alarm transitions into state OK marbot will search for a related alert and close it for you. An auto-closed alert will look like this:

Notify in channel (Slack only)

By default, marbot sends alerts to a channel and adds @here to the message. Doing sends notifications to all active users in the channel.

In case no one acknowledges the alert within five minutes, marbot adds @channel to the message which will send out notifications to active and inactive users of the channel.

Don’t want marbot to add @here and @channel to alerts? Send to following message to the channel to disable the option.

@marbot configure

Next, press the Disable button in the Notify in channel section.

Escalate to individual users

Escalating alerts to individual users is disabled by default. To enable this option post the following message to a channel.

@marbot configure

A dialog to configure the channel’s endpoint appears. Here is what the dialog looks like in Slack.

And here is the same dialog in Microsoft Teams.

Click the Enable button in the Escalate to individual users section.

For all upcoming alerts, escalation happens in four steps when using Slack:

1. Send direct message to an active user.
2. Send direct message to another active user.
3. Send message to channel mention @here.
4. Send message to channel mention @channel.

When using Microsoft Teams, marbot escalates alerts in three steps:

1. Send direct message to an active user.
2. Send direct message to another active user.
3. Send message to channel.