Escalation

Escalations are best explained in a short video. If you prefer written text, scroll down!

Video

Prose

Imagine a Slack team with a #devops channel. Three users and marbot are in the channel:

1. Hanna
2. Nick
3. Ebru
4. marbot

All users are active, which means that they are:

1. online
2. have not enabled Do not Disturb (DND)
3. have not enabled Snooze

Escalation happens in four steps:

1. Send direct message to an active user
2. Send direct message to another active user
3. Send message to channel mention @here
4. Send message to channel mention @channel

Let’s use an example to explain this in details.

Escalation level: First user

Now, an alert is triggered by marbot because an event was sent to an endpoint. marbot will pick on of the active users randomly (in this case Hanna) and sends her a direct message. If no user is active, the next escalation step is triggered. The direct message contains all the relevant information and three buttons.

Four events are possible:

1. Hanna does nothing within five minutes. The escalation continues.
2. Hanna clicks on the Acknowledge button and stops the escalation. She now has time to solve the issue which she can confirm by pressing the Close button.
3. Hanna clicks on the Pass button and continues the escalation.
4. Hanna clicks on the Close button and stops the escalation. She confirms that the issue was resolved.

Hanna did not react in time, that’s why marbot continues to escalate the alert.

Escalation level: Second user

marbot will again pick on of the active users (except Hanna) randomly (in this case Nick) and sends him a direct message. If no user is active, the next escalation step is triggered. The direct message contains all the relevant information and three buttons.

Four events are possible:

1. Nick does nothing within five minutes. The escalation continues.
2. Nick clicks on the Acknowledge button and stops the escalation. He now has time to solve the issue which he can confirm by pressing the Close button.
3. Nick clicks on the Pass button and continues the escalation.
4. Nick clicks on the Close button and stops the escalation. He confirms that the issue was resolved.

Nick clicks the Pass button because he is on a call with a customer and has no time to react.

Escalation level: @here

marbot will send a message to the #devops channel with the @here mention. If no user (except Hanna and Nick) is active, the next escalation step is triggered. This will notify just the team members in the channel who are currently active in Slack. The message contains all the relevant information and two buttons.

Three events are possible:

1. No reaction within five minutes. The escalation continues.
2. A user clicks on the Acknowledge button and stops the escalation. She now has time to solve the issue which she can confirm by pressing the Close button.
3. A user clicks on the Close button and stops the escalation. She confirms that the issue was resolved.

Ebru acknowledges the alert. She receives a direct message from marbot with the option to close the alert.

Escalation level: @channel

marbot will send a message to the #devops channel with the @channel mention. This will notify all team members in the channel. The message contains all the relevant information and two buttons.

Two events are possible:

1. A user clicks on the Acknowledge button and stops the escalation. He now has time to solve the issue which he can confirm by pressing the Close button.
2. A user clicks on the Close button and stops the escalation. He confirms that the issue was resolved.

Configuration

You can configure an endpoint to skip the first user, second user, and @here escalation level. So all alerts show up directly in the channel. You can also disable that marbot announces alerts in channels with @here and @channel.

To configure the endpoint, send a message in a channel mentioning @marbot:

@marbot Configure this endpoint!