Smart alert escalation for Sumo Logic Scheduled Searches for Logs and Monitors for Metrics

Michael Wittig – 09 Feb 2018

Sumo Logic provides a unified platform for all logs and metrics. Analyze all data in real-time and monitor apps and infrastructure in real-time. Scheduled Searches are saved searches that run automatically at specified intervals against your logs and can trigger an action. Monitors watch a time series to alert when the metric has crossed a static threshold. Sumo Logic Scheduled Searches and Monitors also integrate with Slack, but the native integration lacks many features, such as:

With marbot, you get the missing features to manage Sumo Logic Alerts in a team that cares about production.

Set up instructions

First of all, you need to create a WebHook connection to enable Sumo Logic to integrate with marbot.

WebHook connection

  1. Create a free Sumo Logic trial account.
  2. Log in to your Sumo Logic account.
  3. Click on Manage Date at the bottom right and then click on Settings.
  4. Click on Connections at the top.
  5. Click the plus button on the top right.
    Setup Sumo Logic connection: step 1
  6. Select the connection type Webhook.
    Setup Sumo Logic connection: step 2
  7. Set Name to Name.
  8. Set URL to https://api.marbot.io/v1/endpoint/$endpoint-ID. Replace $endpoint-ID with the ID of your endpoint. You can get this value by asking @marbot for it on your Slack channel.
  9. Set Payload to:

    {
    "name": "<$SearchQueryUrl|$SearchName>",
    "description": "$SearchDescription",
    "timeRange": "$TimeRange",
    "logs": "$RawResultsJson",
    "alert": "$AlertSource $AlertThreshold $AlertStatus"
    }
  10. Click on Test Connection to simulate an alert.

  11. Click on Save.
    Setup Sumo Logic connection: step 3

The following alert will be delivered by marbot soon.

Sumo Logic Test Alert

Now, you can create a Scheduled Search or Monitor to send alerts to marbot.

More information: Sumo Logic documentation

  1. Click on New at the top and then click on Log Search.
    Setup Sumo Logic Scheduled Search: step 1
  2. Formulate a search query. E.g., all logs that contain fail*.
  3. Once the query returns the desired results, click on Save as.
    Setup Sumo Logic Scheduled Search: step 2
  4. Set Name to marbot.
  5. Click on Schedule this search.
    Setup Sumo Logic Scheduled Search: step 3
  6. Set Run frequency to Every 15 minutes
  7. Set Alert Type to Webhook.
  8. Set Connection to marbot.
  9. Click on Save.
    Setup Sumo Logic Scheduled Search: step 4

A Scheduled Search alert delivered by marbot will look like this:

Sumo Logic Scheduled Search Alert

More information: Sumo Logic documentation

Read one, if you want to configure which fields are displayed by marbot.

Configure alert fields

Available variables are:

  • $SearchName: Name of the saved search or Monitor.
  • $SearchDescription: Description of the saved search or Monitor.
  • $SearchQuery: The query used to run the saved search.
  • $SearchQueryUrl: The URL link to the search or metrics query.
  • $TimeRange: Time range used to run the search or time range that triggered the metrics alert.
  • $FireTime: The start time of the search or time that metrics alert triggered.
  • $AggregateResultsJson: JSON object containing search aggregation results. (Important: A maximum of 200 results for this field can be sent via Webhook.)
  • $RawResultsJson: JSON object containing raw messages. (Important: Only 10 results for this field can be sent via Webhook.)
  • $NumRawResults: Number of results returned by the search. (Important: Only 100 results for this field can be sent via Webhook.)

Example Payload:

{
"name": "<$SearchQueryUrl|$SearchName>",
"description": "$SearchDescription",
"timeRange": "$TimeRange",
"logs": "$RawResultsJson"
}

More information: Sumo Logic documentation

Monitor

  1. Click on New at the top and then click on Metrics.
    Setup Sumo Logic Monitor: step 1
  2. Formulate a metrics query. E.g., select one metric metric=RequestCounnt.
  3. Keep your mouse over the metrics query and click on the alert icon.
    Setup Sumo Logic Monitor: step 2
  4. Click on Set threshold for critical
  5. Set the threshold to a value that indicates critical system status.
  6. Set Send Notification via to marbot.
  7. Set Monitor Name to marbot.
  8. Click on Save.
    Setup Sumo Logic Monitor: step 3

A Scheduled Search alert delivered by marbot will look like this:

Sumo Logic Monitor Alert

More information: Sumo Logic documentation

Read one, if you want to configure which fields are displayed by marbot.

Configure alert fields

  • $SearchName: Name of the saved search or Monitor.
  • $SearchDescription: Description of the saved search or Monitor.
  • $SearchQuery: The query used to run the saved search.
  • $SearchQueryUrl: The URL link to the search or metrics query.
  • $TimeRange: Time range used to run the search or time range that triggered the metrics alert.
  • $FireTime: The start time of the search or time that metrics alert triggered.
  • $AlertThreshold: The condition that triggered the alert (for example, above 90 at least once in the last 5 minutes).
  • $AlertSource: The metric and sourceHost that triggered the alert.
  • $AlertID: The ID of the triggered alert.
  • $AlertStatus: Current status of the time series that triggered (for example, Critical or Warning).

Example Payload:

{
"name": "<$SearchQueryUrl|$SearchName>",
"description": "$SearchDescription",
"timeRange": "$TimeRange",
"alert": "$AlertSource $AlertThreshold $AlertStatus"
}

More information: Sumo Logic documentation

Michael Wittig

Michael Wittig

Consultant focusing on Amazon Web Services (AWS). Entrepreneur building marbot.io. Author of Amazon Web Services in Action, Rapid Docker on AWS, and cloudonaut.io.

You can contact me via Email, Twitter, and LinkedIn.

Published on

marbot teaser

Chatbot for AWS Monitoring

Configure monitoring for Amazon Web Services: CloudWatch, EC2, RDS, EB, Lambda, and more. Receive and manage alerts via Slack. Solve incidents as a team.

Slack
Add to Slack
Microsoft Teams
Add to Teams