Configuring SinksΒΆ

Robusta can send notifications to various destinations, called sinks.

For a list of all sinks, refer to Sinks Reference.

Defining SinksΒΆ

Sinks are defined in Robusta's Helm chart, using the sinksConfig value:

sinksConfig:
- ms_teams_sink:                  # sink type
    name: my_teams_sink           # arbitrary name
    webhook_url: <placeholder>    # a sink-specific parameter
    match: {}                     # optional routing rules (see below)
    default: true                 # optional (see below)

To add a sink, update sinksConfig according to the instructions in Sinks Reference. Then do a Helm Upgrade.

Configure as many sinks as you like.

Routing Alerts to Specific SinksΒΆ

Define which messages a sink accepts using matchers.

For example, Slack can be configured to receive high-severity messages in a specific namespace. Other messages will not be sent to Slack.

sinksConfig:
- slack_sink:
    name: test_sink
    slack_channel: test-notifications
    api_key: secret-key
    match:
      namespace: [prod]
      severity: [HIGH]

The above example defines match conditions by both namespace and severity. When multiple conditions are present, all must be satisfied.

By default, every message is sent to every matching sink. To change this behaviour, you can mark a sink as non-default.

Matches Can Be Lists Or RegexesΒΆ

Every match rule supports both regular expressions and a list of exact values:

sinksConfig:
- slack_sink:
    name: prod_slack_sink
    slack_channel: prod-notifications
    api_key: secret-key
    # AND between namespace and severity and labels
    match:
      namespace: ^prod$ # match the "prod" namespace exactly
      severity: [HIGH, LOW] # either HIGH or LOW (or logic)
      labels: "foo=bar,instance=123"

The following attributes can be included in a match block:

  • title: e.g. Crashing pod foo in namespace default

  • name : the Kubernetes object name

  • namespace: the Kubernetes object namespace

  • node : the Kubernetes node name

  • severity: one of INFO, LOW, MEDIUM, HIGH

  • type: one of ISSUE, CONF_CHANGE, HEALTH_CHECK, REPORT

  • kind: one of deployment, node, pod, job, daemonset

  • source: one of NONE, KUBERNETES_API_SERVER, PROMETHEUS, MANUAL, CALLBACK

  • identifier: e.g. report_crash_loop

  • labels: A comma separated list of key=val e.g. foo=bar,instance=123

  • annotations: A comma separated list of key=val e.g. app.kubernetes.io/name=prometheus

Note

labels and annotations are both the Kubernetes resource labels and annotations (e.g. pod labels) and the Prometheus alert labels and annotations. If both contains the same label/annotation, the value from the Prometheus alert is preferred.

How do I find the identifier value to use in a match block?

For Prometheus alerts, it's always the alert name.

For custom playbooks, it's the value you set in create_finding under aggregation_key.

Ask us in Slack if you need help.

The regular expressions must be in the Python re module format, as passed to re.match.

Or Between MatchesΒΆ

You can use Or between match rules:

sinksConfig:
- slack_sink:
    name: prod_slack_sink
    slack_channel: prod-notifications
    api_key: secret-key
    # AND between namespace and labels, but or within each selector
    match:
      namespace:
      - default
      - robusta
      labels:
      - "instance=123"
      - "instance=456"

The above will match a resource from namespace (default or robusta) and label (instance=123 or instance=456)

Alternative Routing MethodsΒΆ

For customPlaybooks, there is another option for routing notifications.

Instead of using sink matchers, you can set the sinks attribute per playbook:

customPlaybooks:
- triggers:
  - on_job_failure: {}
  actions:
  - create_finding:
      aggregation_key: "job_failure"
      title: "Job Failed"
  - job_info_enricher: {}
  - job_events_enricher: {}
  - job_pod_enricher: {}
  sinks:
    - "some_sink"
    - "some_other_sink"

Notifications generated this way are sent exclusively to the specified sinks. They will still be filtered by matchers.

Non-Default SinksΒΆ

To prevent a sink from receiving most notifications, you can set default: false. In this case, notifications will be routed to the sink only from customPlaybooks that explicitly name this sink.

Here too, matchers apply as usual and perform further filtering.

ExamplesΒΆ

πŸŽ“ Route Alerts By Namespace

πŸŽ“ Route Alerts By Type

πŸŽ“ Routing with Exclusion Rules

See AlsoΒΆ

πŸ”” All Sinks

πŸŽ“ Silencing Alerts