Routing Alerts To Specific Sinks¶

This guide covers routing notifications to specific sinks by defining a scope.

Sinks in Robusta determine where notifications are sent (e.g. Slack, email). By defining scopes, you can send notifications to different channels based on severity, namespace, and other conditions.

Inclusion Scopes¶

If scope.include is defined, sinks will only receive matching notifications.

For example, to only send notifications with severity HIGH:

sinksConfig:
- slack_sink:
    name: slack_urgent
    slack_channel: urgent
    api_key: secret-key
    scope:
      include:
        - severity: HIGH

For all attributes that can go inside include, refer to Scope Reference.

An undefined include section will effectively include all alerts.

Exclusion Scopes¶

If scope.exclude is defined, sinks will skip matching notifications.

This can be combined with scope.include. For example, we can include notifications from namespace default but ignore specific notifications by name even in that namespace.

sinksConfig:
- slack_sink:
    name: prod_sink
    slack_channel: prod-notifications
    api_key: secret-key
    scope:
      include:
        - namespace: default
      exclude:
        # regardless of the include sections, this will drop all alerts with name CrashLoopBackoff or ImagePullBackoff
        - identifier: [CrashLoopBackoff, ImagePullBackoff]

The general rule is that a notification must match one of the include sections, and must not match all the exclude sections.

You can use the same attributes in exclude as include - refer to Scope Reference.

An undefined exclude section will not exclude anything.

Stop Further Notifications¶

When using multiple sinks, notifications are sent to all matching sinks by default.

To stop sending notifications after a match, set stop: true.

You can use stop to define a fallback sink:

sinksConfig:
- slack_sink:
    name: production_sink
    slack_channel: production-notifications
    api_key: secret-key
    scope:
      include:
        - namespace: production
    stop: true

# because the previous sink sets stop: true, this sink will only receive alerts not matched by the previous sink
- slack_sink:
    name: fallback_sink
    slack_channel: all-other-notifications
    api_key: secret-key

Note that the order of sinks matters! The first sink that matches a notification will receive it. That sink's stop value will determine if further sinks have a chance to receive the notification.

Advanced Scope Conditions¶

Mapping Prometheus Alerts to Kubernetes Resources¶

In Kubernetes environments, Robusta automatically maps Prometheus alerts to Kubernetes resources according to alert labels. This lets you route alerts not only based on Prometheus metric labels, but also based on Kubernetes metadata.

For example, to route a Prometheus KubePodCrashLooping alert based on the related pod's app.kubernetes.io/name label:

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: test-notifications
    api_key: secret-key
    scope:
      include:
        - identifier: "KubePodCrashLooping"
          labels: "app.kubernetes.io/name=my-app"

Note that we routed based on Kubernetes metadata, not present in Prometheus itself!

AND Logic¶

In the following example, we define a sink that matches notifications which are both high-severity and in namespace prod:

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: high-severity-and-prod
    api_key: secret-key
    scope:
      include:
        - namespace: prod
          severity: HIGH

Important: there is no dash character before severity. As we'll see in the next section, this is the difference between AND/OR syntax.

OR Logic¶

To use OR logic, use multiple list elements inside the include block (each element starting with -).

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: high-severity-or-prod
    api_key: secret-key
    scope:
      include:
        # OR between both conditions
        - namespace: prod
        - severity: HIGH

Important: note the dash character in``- severity``. This dash is critical as it separates the two conditions by OR.

"include": [
  {"namespace": "prod", "severity": "HIGH"}
]

"include": [
  {"namespace": "prod"},
  {"severity": "HIGH"}
]

Combining AND/OR Logic¶

You can define complex conditions, such as HIGH severity notifications from staging + all alerts from prod:

sinksConfig:
- slack_sink:
    name: high-severity-staging-or-prod
    slack_channel: high-severity-staging-or-prod
    api_key: secret-key
    scope:
      # define 2 include elements, with an OR between them
      include:
        # this is the first include element - made up of two conditions with AND between them
        - namespace: staging
          severity: HIGH

        # this is the 2nd include element, with a single condition
        - namespace: prod

Matching Lists of Values¶

The following will match any value in the list (either prod or default):

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: test-notifications
    api_key: secret-key
    scope:
      include:
        - namespace: [prod, default]

This is equivalent to:

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: test-notifications
    api_key: secret-key
    scope:
      include:
        - namespace: prod
        - namespace: default

Using Regexes¶

You can also use regexes:

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: test-notifications
    api_key: secret-key
    scope:
      include:
        # this will match kube-system
        - namespace: kube-.*

Kubernetes Label Selectors¶

You can match on Kubernetes label selectors with special syntax:

sinksConfig:
- slack_sink:
    name: prod_slack_sink
    slack_channel: prod-notifications
    api_key: secret-key
    scope:
      include:
        # label selectors are interpreted like Kubernetes - selectors separated by comma are ANDED together
        - labels: "instance=1,foo!=x.*"

Testing Alert Routing¶

Scope Reference¶

Here is the complete list of attributes that can be used in include / exclude sections:

When processing the scope block, the following rules apply:

1. If the notification is excluded by any of the sink scope excludes - drop it

2. If the notification is included by any of the sink scope includes - accept it

3. If the notification is included by any of the sink matchers (deprecated) - accept it

Alternative Routing Methods¶

For customPlaybooks, there is another option for routing notifications.

Instead of using sink matchers, you can set the sinks attribute per playbook:

customPlaybooks:
- triggers:
  - on_job_failure: {}
  actions:
  - create_finding:
      aggregation_key: "JobFailure"
      title: "Job Failed"
  - job_info_enricher: {}
  - job_events_enricher: {}
  - job_pod_enricher: {}
  sinks:
    - "some_sink"
    - "some_other_sink"

Notifications generated this way are sent exclusively to the specified sinks. They will still be filtered by matchers.

If you use this method, you can set default: false in the sink definition and it will be ignored for all notifications except those from custom playbooks that explicitly name this sink.