AlertManager and Prometheus Integration¶
Sending Alerts to Robusta¶
For Robusta to improve Prometheus alerts, Robusta has to first receive those alerts from AlertManager.
If you installed Robusta's Embedded Prometheus Stack then no configuration is necessary. For other setups, read on!
To configure Prometheus to send alerts to Robusta, add two settings to AlertManager:
A webhook receiver for Robusta
A route for the webhook receiver you added
Below is an example AlertManager configuration. Depending on your setup, the exact file to edit may vary. (See below.)
AlertManager config for sending alerts to Robusta
receivers: - name: 'robusta' webhook_configs: # the following line assumes that Robusta was installed in the `default` namespace. # if you installed Robusta in a different namespace, replace `default` with the correct namespace # likewise, if you named your Helm release ``robert`` then replace ``robusta`` with ``robert`` - url: 'http://robusta-runner.default.svc.cluster.local/api/alerts' send_resolved: true route: routes: - receiver: 'robusta' matchers: - severity =~ "info|warn|error|critical" repeat_interval: 4h continue: true
Make sure the Robusta
routeis the first
routedefined. If it isn't the first route, it might not receive alerts. When a
routeis matched, the alert will not be sent to following routes, unless the
routeis configured with
- Tweak the settings accordingly if:
You installed Robusta in a namespace other than
You named Robusta's Helm release something other than
After you configure AlertManager, you can test it works properly, by creating a demo alert:
Within a few minutes, you should see the demo alert in the Robusta UI, Slack, and any other sinks you configured.
Why do I still see a banner in the Robusta UI that "Alerts won't show up"?
The notification is displayed until the first alert is sent from AlertManager to Robusta.
Here are instructions for configuring AlertManager in specific setups. Don't see your setup? Just follow the General Instructions above.
kube-prometheus-stack and Prometheus Operator¶
If you installed kube-prometheus-stack or the Prometheus Operator by yourself (not via Robusta) then tell AlertManager about Robusta using a Kubernetes Secret. The Prometheus Operator will pass this secret to AlertManager, which will then push alerts to Robusta by webhook.
To configure the secret, copy the configuration here and place it in the appropriate secret.
Why use a secret instead of editing AlertManagerConfig?
In theory, you can configure an AlertmanagerConfig instead of using a secret. However, this is not recommended. It will only forward alerts from one namespace.
Out-of-cluster Prometheus Installations¶
If AlertManager is located outside of your Kubernetes cluster then a few more steps are necessary:
Enable two-way interactivity in Robusta's configuration by setting
Make sure that your alerts contain a label named
cluster_namewhich matches the cluster_name defined in Robusta's configuration. This is necessary so that the Robusta cloud knows which cluster to forward events to.
Configure AlertManager as follows:
receivers: - name: 'robusta' webhook_configs: - url: 'https://api.robusta.dev/integrations/generic/alertmanager' http_config: authorization: # Replace <TOKEN> with a string in the format `<ACCOUNT_ID> <SIGNING_KEY>` credentials: <TOKEN> send_resolved: true route: routes: - receiver: 'robusta' matchers: - severity =~ "info|warn|error|critical" repeat_interval: 4h continue: true
Robusta's Embedded Prometheus¶
If you installed Robusta's Embedded Prometheus Stack then no configuration is necessary.