AlertManager and Prometheus Integration

Sending Alerts to Robusta

For Robusta to improve Prometheus alerts, Robusta has to first receive those alerts from AlertManager.

If you installed Robusta's Embedded Prometheus Stack then no configuration is necessary. For other setups, read on!

General Instructions

To configure Prometheus to send alerts to Robusta, add two settings to AlertManager:

  1. A webhook receiver for Robusta

  2. A route for the webhook receiver you added

Below is an example AlertManager configuration. Depending on your setup, the exact file to edit may vary. (See below.)

AlertManager config for sending alerts to Robusta

receivers:
  - name: 'robusta'
    webhook_configs:
      # the following line assumes that Robusta was installed in the `default` namespace.
      # if you installed Robusta in a different namespace, replace `default` with the correct namespace
      # likewise, if you named your Helm release ``robert`` then replace ``robusta`` with ``robert``
      - url: 'http://robusta-runner.default.svc.cluster.local/api/alerts'
        send_resolved: true

route:
  routes:
    - receiver: 'robusta'
      matchers:
        - severity =~ "info|warn|error|critical"
      repeat_interval: 4h
      continue: true

Common Mistakes

  1. Make sure the Robusta route is the first route defined. If it isn't the first route, it might not receive alerts. When a route is matched, the alert will not be sent to following routes, unless the route is configured with continue: true.

  2. Tweak the settings accordingly if:
    • You installed Robusta in a namespace other than default

    • You named Robusta's Helm release something other than robusta

After you configure AlertManager, you can test it works properly, by creating a demo alert:

robusta demo-alert

Within a few minutes, you should see the demo alert in the Robusta UI, Slack, and any other sinks you configured.

Why do I still see a banner in the Robusta UI that "Alerts won't show up"?

The notification is displayed until the first alert is sent from AlertManager to Robusta.

Specific Instructions

Here are instructions for configuring AlertManager in specific setups. Don't see your setup? Just follow the General Instructions above.

kube-prometheus-stack and Prometheus Operator

If you installed kube-prometheus-stack or the Prometheus Operator by yourself (not via Robusta) then tell AlertManager about Robusta using a Kubernetes Secret. The Prometheus Operator will pass this secret to AlertManager, which will then push alerts to Robusta by webhook.

To configure the secret, copy the configuration here and place it in the appropriate secret.

Why use a secret instead of editing AlertManagerConfig?

In theory, you can configure an AlertmanagerConfig instead of using a secret. However, this is not recommended. It will only forward alerts from one namespace.

Out-of-cluster Prometheus Installations

If AlertManager is located outside of your Kubernetes cluster then a few more steps are necessary:

  1. Enable two-way interactivity in Robusta's configuration by setting disableCloudRouting: false

  2. Make sure that your alerts contain a label named cluster_name which matches the cluster_name defined in Robusta's configuration. This is necessary so that the Robusta cloud knows which cluster to forward events to.

  3. Configure AlertManager as follows:

alertmanager.yaml

receivers:
  - name: 'robusta'
    webhook_configs:
      - url: 'https://api.robusta.dev/integrations/generic/alertmanager'
        http_config:
          authorization:
            # Replace <TOKEN> with a string in the format `<ACCOUNT_ID> <SIGNING_KEY>`
            credentials: <TOKEN>
        send_resolved: true

route:
  routes:
  - receiver: 'robusta'
    matchers:
      - severity =~ "info|warn|error|critical"
    repeat_interval: 4h
    continue: true

Robusta's Embedded Prometheus

If you installed Robusta's Embedded Prometheus Stack then no configuration is necessary.