Configuring SinksΒΆ
Robusta can send notifications to various destinations, called sinks.
For a list of all sinks, refer to Sinks Reference.
Defining SinksΒΆ
Sinks are defined in Robusta's Helm chart, using the sinksConfig
value:
sinksConfig:
- ms_teams_sink: # sink type
name: my_teams_sink # arbitrary name
webhook_url: <placeholder> # a sink-specific parameter
match: {} # optional routing rules (see below)
default: true # optional (see below)
To add a sink, update sinksConfig
according to the instructions in Sinks Reference. Then do a Helm Upgrade.
Configure as many sinks as you like.
Routing Alerts to Specific SinksΒΆ
Define which messages a sink accepts using matchers.
For example, Slack can be configured to receive high-severity messages in a specific namespace. Other messages will not be sent to Slack.
sinksConfig:
- slack_sink:
name: test_sink
slack_channel: test-notifications
api_key: secret-key
match:
namespace: [prod]
severity: [HIGH]
The above example defines match conditions by both namespace
and severity
. When multiple conditions are present, all must be satisfied.
By default, every message is sent to every matching sink. To change this behaviour, you can mark a sink as non-default.
Matches Can Be Lists Or RegexesΒΆ
Every match rule supports both regular expressions and a list of exact values:
sinksConfig:
- slack_sink:
name: prod_slack_sink
slack_channel: prod-notifications
api_key: secret-key
# AND between namespace and severity and labels
match:
namespace: ^prod$ # match the "prod" namespace exactly
severity: [HIGH, LOW] # either HIGH or LOW (or logic)
labels: "foo=bar,instance=123"
The following attributes can be included in a match block:
title
: e.g.Crashing pod foo in namespace default
name
: the Kubernetes object namenamespace
: the Kubernetes object namespacenode
: the Kubernetes node nameseverity
: one ofINFO
,LOW
,MEDIUM
,HIGH
type
: one ofISSUE
,CONF_CHANGE
,HEALTH_CHECK
,REPORT
kind
: one ofdeployment
,node
,pod
,job
,daemonset
source
: one ofNONE
,KUBERNETES_API_SERVER
,PROMETHEUS
,MANUAL
,CALLBACK
identifier
: e.g.report_crash_loop
labels
: A comma separated list ofkey=val
e.g.foo=bar,instance=123
annotations
: A comma separated list ofkey=val
e.g.app.kubernetes.io/name=prometheus
Note
labels
and annotations
are both the Kubernetes resource labels and annotations (e.g. pod labels) and the Prometheus alert labels and annotations.
If both contains the same label/annotation, the value from the Prometheus alert is preferred.
How do I find the identifier
value to use in a match block?
For Prometheus alerts, it's always the alert name.
For custom playbooks, it's the value you set in create_finding under aggregation_key
.
Ask us in Slack if you need help.
The regular expressions must be in the Python re module format, as passed to re.match.
Or Between MatchesΒΆ
You can use Or between match rules:
sinksConfig:
- slack_sink:
name: prod_slack_sink
slack_channel: prod-notifications
api_key: secret-key
# AND between namespace and labels, but or within each selector
match:
namespace:
- default
- robusta
labels:
- "instance=123"
- "instance=456"
The above will match a resource from namespace (default or robusta) and label (instance=123 or instance=456)
Alternative Routing MethodsΒΆ
For customPlaybooks, there is another option for routing notifications.
Instead of using sink matchers, you can set the sinks attribute per playbook:
customPlaybooks:
- triggers:
- on_job_failure: {}
actions:
- create_finding:
aggregation_key: "job_failure"
title: "Job Failed"
- job_info_enricher: {}
- job_events_enricher: {}
- job_pod_enricher: {}
sinks:
- "some_sink"
- "some_other_sink"
Notifications generated this way are sent exclusively to the specified sinks. They will still be filtered by matchers.
Non-Default SinksΒΆ
To prevent a sink from receiving most notifications, you can set default: false
. In this case, notifications will be
routed to the sink only from customPlaybooks that explicitly name this sink.
Here too, matchers apply as usual and perform further filtering.
ExamplesΒΆ
π Route Alerts By Namespace
π Route Alerts By Type
π Routing with Exclusion Rules
See AlsoΒΆ
π All Sinks
π Silencing Alerts