Route Alerts By TypeΒΆ

By default, all Robusta notifications are sent to all sinks.

It can be useful to send alerts of a specific type to a dedicated channel.

In this guide, we'll show how to route notifications for crashing pods to a specific Slack channel. All other notifications will be sent to the usual channel.

PrerequisitesΒΆ

All least one existing sink must be configured.

Setting Up RoutingΒΆ

This guide applies to all sink types. For simplicity's sake we'll assume you have an existing Slack sink:

sinksConfig:
- slack_sink:
    name: slack_app_sink
    slack_channel: main-notifications
    api_key: secret-key

The first step is to duplicate your sink. You need two unique sinks - one for each channel:

sinksConfig:
- slack_sink:
    name: main_sink
    slack_channel: main-notifications
    api_key: secret-key
- slack_sink:
    name: crashloopbackoff_slack_sink
    slack_channel: crashpod-notifications
    api_key: secret-key

The sinks are nearly identical - only the name and slack_channel parameters vary.

Now lets add a matcher to each sink, so it receives a subset of notifications:

 sinksConfig:
 - slack_sink:
     name: main_sink
     slack_channel: main-notifications
     api_key: secret-key
     - scope:
         exclude:
         # don't send notifications related to image pull backoff
           - identifier: [image_pull_backoff_reporter]

- slack_sink:
     name: crashloopbackoff_slack_sink
     slack_channel: crash-notifications
     api_key: secret-key
     - scope:
         include:
         # only send notifications related to crashing pods and CPU throttling
           - identifier: [report_crash_loop, CPUThrottlingHigh]

Note

For Prometheus alerts use the Alert name. Example, CPUThrottlingHigh, KubeContainerWaiting. For other events, use the name as it appears on the Robusta timeline. Example, report_crash_loop and image_pull_backoff_reporter

Now the crash-notifications channel will receive crashpod notifications and all other notifications will go to the main-notifications channel.

Further ReadingΒΆ